clickhouse-airbyte-normer

A lightweight Go tool to transform Airbyte raw tables into normalized tables in ClickHouse for easier querying.

Overview

This tool is a quick and simple solution to migrate raw data from Airbyte into structured tables, making it more accessible for analytics. It's not production-ready, but it helps improve data usability with minimal setup.

Environment Variables

Set these environment variables when running the tool in a Docker container:

Variable	Description
CH_HOST	ClickHouse host (e.g., clickhouse.local)
CH_PORT	ClickHouse port (default: 9000)
CH_USER	ClickHouse user with sufficient privileges
CH_PASS	Password for the ClickHouse user
RAW_DB	Database where Airbyte stores raw data

Example configuration:

CH_HOST="clickhouse.local"
CH_PORT="9000"
CH_USER="default"
CH_PASS="password"
RAW_DB="airbyte_raw"

Flags:

The tool supports the following flags:

Flag	Description
-force-once	Runs the migration immediately instead of waiting for a webhook trigger
-debug	Enables debug logging for troubleshooting
-drop-raw	Drops raw tables after migration (default: keeps raw tables)

Usage example:

./clickhouse-airbyte-normer -force-once -debug

And in airbyte set the webhook to hostname:8080/webhook on successfull sync

Notes

This is a quick hack and not a production-ready solution. Use at your own discretion, especially when enabling -drop-raw. May cause dataloss Contributions and improvements are welcome! (on github)