diff --git a/README.md b/README.md index a799b97..2f6c79c 100644 --- a/README.md +++ b/README.md @@ -1,32 +1,51 @@ -# clickhouse-airbyte-normer +# clickhouse-airbyte-normer -GoProg to move data from raw tables to normal tables #hack +A lightweight Go tool to transform Airbyte raw tables into normalized tables in ClickHouse for easier querying. -A quick hack to get the Airbyte raw tables to normal tables so my collegueas can query the data better in clickhouse. +## Overview -env variables for docker build: +This tool is a quick and simple solution to migrate raw data from Airbyte into structured tables, making it more accessible for analytics. It's not production-ready, but it helps improve data usability with minimal setup. -``` - chHost := os.Getenv("CH_HOST") - chPort := os.Getenv("CH_PORT") - chUser := os.Getenv("CH_USER") - chPass := os.Getenv("CH_PASS") - rawDB := os.Getenv("RAW_DB") +## Environment Variables + +Set these environment variables when running the tool in a Docker container: + +| Variable | Description | +|-----------|------------| +| `CH_HOST` | ClickHouse host (e.g., `clickhouse.local`) | +| `CH_PORT` | ClickHouse port (default: `9000`) | +| `CH_USER` | ClickHouse user with sufficient privileges | +| `CH_PASS` | Password for the ClickHouse user | +| `RAW_DB` | Database where Airbyte stores raw data | + +Example configuration: + +```sh +CH_HOST="clickhouse.local" +CH_PORT="9000" +CH_USER="default" +CH_PASS="password" +RAW_DB="airbyte_raw" ``` -eg: -CH_HOST: clickhouse.local -CH_PORT: 9000 (native clickhouse) -CH_USER: default (specifically a user with just enough rights) -CH_PASS: password -RAW_DB: Raw database that clickhouse writes to +Flags: -flags +The tool supports the following flags: -``` - forceOnce := flag.Bool("force-once", false, "Run the migration immediately without waiting for a webhook") - debug := flag.Bool("debug", false, "Enable debug logging") - dropRawFlag := flag.Bool("drop-raw", false, "Drop raw tables after migration (default: keep raw tables)") -``` +| Flag | Description| +|-------|------------| +|-force-once | Runs the migration immediately instead of waiting for a webhook trigger | +|-debug Enables | debug logging for troubleshooting | +|-drop-raw | Drops raw tables after migration (default: keeps raw tables) | -It's not production ready but feel free to use as a quick hack to at least make the data more usable. \ No newline at end of file +Usage example: + +`./clickhouse-airbyte-normer -force-once -debug` + +And in airbyte set the webhook to hostname:8080/webhook on successfull sync + +Notes + +This is a quick hack and not a production-ready solution. +Use at your own discretion, especially when enabling -drop-raw. *May cause dataloss* +Contributions and improvements are welcome! (on github) \ No newline at end of file