smo/clickhouse-airbyte-normer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix fast readme file

Browse Source
This commit is contained in:
seppdroid
2025-02-21 19:16:17 +00:00
parent 8a587dcd9b
commit 9c5b2312fe
1 changed files with 42 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
README.md
View File

@@ -1,32 +1,51 @@
# clickhouse-airbyte-normer
# clickhouse-airbyte-normer
GoProg to move data from raw tables to normal tables #hack
A lightweight Go tool to transform Airbyte raw tables into normalized tables in ClickHouse for easier querying.
A quick hack to get the Airbyte raw tables to normal tables so my collegueas can query the data better in clickhouse.
## Overview
env variables for docker build:
This tool is a quick and simple solution to migrate raw data from Airbyte into structured tables, making it more accessible for analytics. It's not production-ready, but it helps improve data usability with minimal setup.
```
chHost := os.Getenv("CH_HOST")
chPort := os.Getenv("CH_PORT")
chUser := os.Getenv("CH_USER")
chPass := os.Getenv("CH_PASS")
rawDB := os.Getenv("RAW_DB")
## Environment Variables
Set these environment variables when running the tool in a Docker container:
| Variable | Description |
|-----------|------------|
| `CH_HOST` | ClickHouse host (e.g., `clickhouse.local`) |
| `CH_PORT` | ClickHouse port (default: `9000`) |
| `CH_USER` | ClickHouse user with sufficient privileges |
| `CH_PASS` | Password for the ClickHouse user |
| `RAW_DB` | Database where Airbyte stores raw data |
Example configuration:
```sh
CH_HOST="clickhouse.local"
CH_PORT="9000"
CH_USER="default"
CH_PASS="password"
RAW_DB="airbyte_raw"
```
eg:
CH_HOST: clickhouse.local
CH_PORT: 9000 (native clickhouse)
CH_USER: default (specifically a user with just enough rights)
CH_PASS: password
RAW_DB: Raw database that clickhouse writes to
Flags:
flags
The tool supports the following flags:
```
forceOnce := flag.Bool("force-once", false, "Run the migration immediately without waiting for a webhook")
debug := flag.Bool("debug", false, "Enable debug logging")
dropRawFlag := flag.Bool("drop-raw", false, "Drop raw tables after migration (default: keep raw tables)")
```
| Flag | Description|
|-------|------------|
|-force-once | Runs the migration immediately instead of waiting for a webhook trigger |
|-debug Enables | debug logging for troubleshooting |
|-drop-raw | Drops raw tables after migration (default: keeps raw tables) |
It's not production ready but feel free to use as a quick hack to at least make the data more usable.
Usage example:
`./clickhouse-airbyte-normer -force-once -debug`
And in airbyte set the webhook to hostname:8080/webhook on successfull sync
Notes
This is a quick hack and not a production-ready solution.
Use at your own discretion, especially when enabling -drop-raw. *May cause dataloss*
Contributions and improvements are welcome! (on github)