Files
clickhouse-airbyte-normer/README.md
2025-02-21 19:16:17 +00:00

1.6 KiB

clickhouse-airbyte-normer

A lightweight Go tool to transform Airbyte raw tables into normalized tables in ClickHouse for easier querying.

Overview

This tool is a quick and simple solution to migrate raw data from Airbyte into structured tables, making it more accessible for analytics. It's not production-ready, but it helps improve data usability with minimal setup.

Environment Variables

Set these environment variables when running the tool in a Docker container:

Variable Description
CH_HOST ClickHouse host (e.g., clickhouse.local)
CH_PORT ClickHouse port (default: 9000)
CH_USER ClickHouse user with sufficient privileges
CH_PASS Password for the ClickHouse user
RAW_DB Database where Airbyte stores raw data

Example configuration:

CH_HOST="clickhouse.local"
CH_PORT="9000"
CH_USER="default"
CH_PASS="password"
RAW_DB="airbyte_raw"

Flags:

The tool supports the following flags:

Flag Description
-force-once Runs the migration immediately instead of waiting for a webhook trigger
-debug Enables debug logging for troubleshooting
-drop-raw Drops raw tables after migration (default: keeps raw tables)

Usage example:

./clickhouse-airbyte-normer -force-once -debug

And in airbyte set the webhook to hostname:8080/webhook on successfull sync

Notes

This is a quick hack and not a production-ready solution. Use at your own discretion, especially when enabling -drop-raw. May cause dataloss Contributions and improvements are welcome! (on github)