# clickhouse-airbyte-normer  

A lightweight Go tool to transform Airbyte raw tables into normalized tables in ClickHouse for easier querying.  

## Overview  

This tool is a quick and simple solution to migrate raw data from Airbyte into structured tables, making it more accessible for analytics. It's not production-ready, but it helps improve data usability with minimal setup.  

## Environment Variables  

Set these environment variables when running the tool in a Docker container:  

| Variable  | Description |
|-----------|------------|
| `CH_HOST` | ClickHouse host (e.g., `clickhouse.local`) |
| `CH_PORT` | ClickHouse port (default: `9000`) |
| `CH_USER` | ClickHouse user with sufficient privileges |
| `CH_PASS` | Password for the ClickHouse user |
| `RAW_DB`  | Database where Airbyte stores raw data |

Example configuration:  

```sh
CH_HOST="clickhouse.local"
CH_PORT="9000"
CH_USER="default"
CH_PASS="password"
RAW_DB="airbyte_raw"
```

Flags:

The tool supports the following flags:

| Flag	| Description|
|-------|------------|
|-force-once |	Runs the migration immediately instead of waiting for a webhook trigger |
|-debug |	Enables debug logging for troubleshooting |
|-drop-raw |	Drops raw tables after migration (default: keeps raw tables) |

Usage example:

`./clickhouse-airbyte-normer -force-once -debug`

And in airbyte set the webhook to hostname:8080/webhook on successfull sync 

Notes

This is a quick hack and not a production-ready solution.
Use at your own discretion, especially when enabling -drop-raw. *May cause dataloss*
Contributions and improvements are welcome! (on github)