51 lines
1.6 KiB
Markdown
51 lines
1.6 KiB
Markdown
# clickhouse-airbyte-normer
|
|
|
|
A lightweight Go tool to transform Airbyte raw tables into normalized tables in ClickHouse for easier querying.
|
|
|
|
## Overview
|
|
|
|
This tool is a quick and simple solution to migrate raw data from Airbyte into structured tables, making it more accessible for analytics. It's not production-ready, but it helps improve data usability with minimal setup.
|
|
|
|
## Environment Variables
|
|
|
|
Set these environment variables when running the tool in a Docker container:
|
|
|
|
| Variable | Description |
|
|
|-----------|------------|
|
|
| `CH_HOST` | ClickHouse host (e.g., `clickhouse.local`) |
|
|
| `CH_PORT` | ClickHouse port (default: `9000`) |
|
|
| `CH_USER` | ClickHouse user with sufficient privileges |
|
|
| `CH_PASS` | Password for the ClickHouse user |
|
|
| `RAW_DB` | Database where Airbyte stores raw data |
|
|
|
|
Example configuration:
|
|
|
|
```sh
|
|
CH_HOST="clickhouse.local"
|
|
CH_PORT="9000"
|
|
CH_USER="default"
|
|
CH_PASS="password"
|
|
RAW_DB="airbyte_raw"
|
|
```
|
|
|
|
Flags:
|
|
|
|
The tool supports the following flags:
|
|
|
|
| Flag | Description|
|
|
|-------|------------|
|
|
|-force-once | Runs the migration immediately instead of waiting for a webhook trigger |
|
|
|-debug | Enables debug logging for troubleshooting |
|
|
|-drop-raw | Drops raw tables after migration (default: keeps raw tables) |
|
|
|
|
Usage example:
|
|
|
|
`./clickhouse-airbyte-normer -force-once -debug`
|
|
|
|
And in airbyte set the webhook to hostname:8080/webhook on successfull sync
|
|
|
|
Notes
|
|
|
|
This is a quick hack and not a production-ready solution.
|
|
Use at your own discretion, especially when enabling -drop-raw. *May cause dataloss*
|
|
Contributions and improvements are welcome! (on github) |