smo/clickhouse-airbyte-normer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix fast readme file

Browse Source
This commit is contained in:
seppdroid
2025-02-21 19:16:17 +00:00
parent 8a587dcd9b
commit 9c5b2312fe
1 changed files with 42 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
README.md
View File

@@ -1,32 +1,51 @@
# clickhouse-airbyte-normer # clickhouse-airbyte-normer
GoProg to move data from raw tables to normal tables #hack A lightweight Go tool to transform Airbyte raw tables into normalized tables in ClickHouse for easier querying.
A quick hack to get the Airbyte raw tables to normal tables so my collegueas can query the data better in clickhouse. ## Overview
env variables for docker build: This tool is a quick and simple solution to migrate raw data from Airbyte into structured tables, making it more accessible for analytics. It's not production-ready, but it helps improve data usability with minimal setup.
``` ## Environment Variables
chHost := os.Getenv("CH_HOST")
chPort := os.Getenv("CH_PORT") Set these environment variables when running the tool in a Docker container:
chUser := os.Getenv("CH_USER")
chPass := os.Getenv("CH_PASS") | Variable | Description |
rawDB := os.Getenv("RAW_DB") |-----------|------------|
| `CH_HOST` | ClickHouse host (e.g., `clickhouse.local`) |
| `CH_PORT` | ClickHouse port (default: `9000`) |
| `CH_USER` | ClickHouse user with sufficient privileges |
| `CH_PASS` | Password for the ClickHouse user |
| `RAW_DB` | Database where Airbyte stores raw data |
Example configuration:
```sh
CH_HOST="clickhouse.local"
CH_PORT="9000"
CH_USER="default"
CH_PASS="password"
RAW_DB="airbyte_raw"
``` ```
eg: Flags:
CH_HOST: clickhouse.local
CH_PORT: 9000 (native clickhouse)
CH_USER: default (specifically a user with just enough rights)
CH_PASS: password
RAW_DB: Raw database that clickhouse writes to
flags The tool supports the following flags:
``` | Flag | Description|
forceOnce := flag.Bool("force-once", false, "Run the migration immediately without waiting for a webhook") |-------|------------|
debug := flag.Bool("debug", false, "Enable debug logging") |-force-once | Runs the migration immediately instead of waiting for a webhook trigger |
dropRawFlag := flag.Bool("drop-raw", false, "Drop raw tables after migration (default: keep raw tables)") |-debug Enables | debug logging for troubleshooting |
``` |-drop-raw | Drops raw tables after migration (default: keeps raw tables) |
It's not production ready but feel free to use as a quick hack to at least make the data more usable. Usage example:
`./clickhouse-airbyte-normer -force-once -debug`
And in airbyte set the webhook to hostname:8080/webhook on successfull sync
Notes
This is a quick hack and not a production-ready solution.
Use at your own discretion, especially when enabling -drop-raw. *May cause dataloss*
Contributions and improvements are welcome! (on github)