pg-avalanche

Replicates Postgres to Iceberg

🌟 Data Platform 🌟

Avalanche 2.0

🦋 Prerequisite

• Docker Installed

• Vault Installed

🛠️ Installation Steps

1. Pull down the Debezium 2 Iceberg Server

make pull-iceberg

2. Ensure you're under data_eng@dev role

role login
role use data_eng@dev

3. Build and run the compose. This takes awhile on the first run since we have to build the builder image first and its not yet cached

make run

4. SSH into the Postgres container and fire whatever commands once logged in

docker exec -it <container id> psql -d postgres -U admin

Notes

1. We added permissions to the confluent user to be able to do all operations on an s3 bucket
2. We added in a custom namespace to the application.properties so we could leverage pgoutput -> debezium.source.plugin.name=pgoutput
3. Brian needs to add the AWS_SECRET_ACCESS_KEY= and AWS_ACCESS_KEY= to vault but his permissions are broken atm so we need to request the values from him atm