diff --git a/content/embeds/rdi-supported-source-versions.md b/content/embeds/rdi-supported-source-versions.md index 815e599d76..7498fc08b7 100644 --- a/content/embeds/rdi-supported-source-versions.md +++ b/content/embeds/rdi-supported-source-versions.md @@ -11,4 +11,4 @@ | AlloyDB for PostgreSQL | 14.2, 15.7 | - | 14.2, 15.7 | | AWS Aurora/PostgreSQL | 15 | 15 | - | | Neon | 14, 15, 16, 17 | - | - | -| Snowflake (preview) | - | - | - | \ No newline at end of file +| Snowflake (preview) | - | - | - | diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md index 05cbc732b9..a7d6515781 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql.md @@ -1,5 +1,5 @@ --- -Title: Prepare PostgreSQL/Supabase for RDI +Title: Prepare PostgreSQL for RDI aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/postgresql/ alwaysopen: false categories: @@ -7,21 +7,15 @@ categories: - integrate - rs - rdi -description: Prepare PostgreSQL databases (including Supabase) to work with RDI +description: Prepare PostgreSQL databases to work with RDI group: di -linkTitle: Prepare PostgreSQL/Supabase +linkTitle: Prepare PostgreSQL summary: Redis Data Integration keeps Redis in sync with the primary database in near real time. type: integration weight: 2 --- -{{< note >}} -[Supabase](https://supabase.com/docs/guides/database/overview) uses PostgreSQL as -its database engine, so the instructions below also apply to Supabase. However, RDI -doesn't currently support cloud deployments of Supabase with AWS or GCP. -{{< /note >}} - PostgreSQL supports several [logical decoding plug-ins](https://wiki.postgresql.org/wiki/Logical_Decoding_Plugins) to enable CDC. If you don't want to use the native `pgoutput` logical replication stream support diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md new file mode 100644 index 0000000000..bcc2e31454 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md @@ -0,0 +1,234 @@ +--- +Title: Prepare Supabase for RDI +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Prepare a hosted Supabase database to work with RDI +group: di +linkTitle: Prepare Supabase +summary: Configure a hosted Supabase PostgreSQL database for snapshot and change data capture with Redis Data Integration. +type: integration +weight: 11 +--- + +[Supabase](https://supabase.com/docs/guides/database/overview) is a hosted +PostgreSQL platform. RDI can connect to a hosted Supabase project +through any direct PostgreSQL endpoint as long as it is reachable from the RDI +deployment and supports logical replication. + +{{< note >}} +RDI supports hosted Supabase projects running an +[RDI-supported PostgreSQL version]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}). +The integration was validated with RDI 1.19.0 and hosted Supabase PostgreSQL +17.6. For self-hosted Supabase deployments, follow the general +[PostgreSQL preparation guide]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql" >}}). +This page describes Supabase setup for a self-managed RDI deployment. For the +managed service, see +[Use Supabase with RDI on Redis Cloud]({{< relref "/operate/rc/rdi/supabase" >}}). +{{< /note >}} + +Supabase differs from a typical self-managed PostgreSQL source in the following +ways: + +- You can't edit `postgresql.conf` or `pg_hba.conf` directly. Supabase enables + logical replication and manages these settings for you. +- You must use the direct database endpoint for logical replication because + [Supavisor connection pooler endpoints don't support logical replication](https://supabase.com/docs/guides/database/replication/manual-replication-faq#which-connection-string-should-be-used). +- The direct endpoint uses IPv6 unless you enable the Supabase dedicated IPv4 + add-on. Enable the add-on if your RDI deployment can't connect over IPv6. +- Supabase can enforce TLS and provides a CA certificate that RDI can use to + validate the database certificate. +- Supabase Row Level Security (RLS) can restrict the rows visible during the + initial snapshot. + +The following checklist summarizes the setup: + +```checklist {id="supabaselist"} +- [ ] [Create or select a Supabase project](#1-create-or-select-a-supabase-project) +- [ ] [Configure direct network access](#2-configure-direct-network-access) +- [ ] [Create a dedicated RDI role](#3-create-a-dedicated-rdi-role) +- [ ] [Grant access to source tables](#4-grant-access-to-source-tables) +- [ ] [Configure TLS](#5-configure-tls) +- [ ] [Configure RDI](#6-configure-rdi) +- [ ] [Monitor replication slots](#7-monitor-replication-slots) +``` + +## 1. Create or select a Supabase project + +Create a project in the [Supabase dashboard](https://supabase.com/dashboard) +or select an existing project. You can find its PostgreSQL version in the +Supabase dashboard or run the following query in the SQL editor: + +```sql +SELECT version(); +``` + +## 2. Configure direct network access + +For a public connection, select **Connect** in the Supabase dashboard and copy +the **Direct connection** hostname. It has the following form: + +```text +db..supabase.co +``` + +You should generally use port `5432`, but you can use a private hostname or +address instead if you have configured private connectivity between the RDI +deployment and Supabase. +Don't use a Supavisor transaction or session pooler connection string because +these endpoints don't support logical replication. + +Supabase direct connections use IPv6 by default. If your RDI deployment +doesn't have IPv6 egress, enable the +[dedicated IPv4 add-on](https://supabase.com/docs/guides/platform/ipv4-address). +The add-on requires a paid Supabase plan. + +If you use the public endpoint and enable +[Supabase Network Restrictions](https://supabase.com/docs/guides/platform/network-restrictions), +add the public egress address of the RDI host or cluster to the allowlist. Use +a `/32` CIDR for an individual IPv4 address. For private connectivity, make +sure the RDI host or cluster can resolve and route to the private endpoint. + +## 3. Create a dedicated RDI role + +In the Supabase SQL editor, create a dedicated login for RDI. Replace the +example name and password with your own values: + +```sql +CREATE ROLE rdi_replication + WITH LOGIN REPLICATION PASSWORD ''; +``` + +{{< warning >}} +Don't use the Supabase `postgres` administrator account for the RDI connection. +The RDI role's credentials provide continuous access to captured data, so grant +the role only the permissions it needs. +{{< /warning >}} + +## 4. Grant access to source tables + +The RDI role needs to connect to the database and read every table included in +the initial snapshot. For example: + +```sql +GRANT CONNECT ON DATABASE postgres TO rdi_replication; + +GRANT USAGE ON SCHEMA public TO rdi_replication; +GRANT SELECT ON ALL TABLES IN SCHEMA public TO rdi_replication; + +ALTER DEFAULT PRIVILEGES IN SCHEMA public + GRANT SELECT ON TABLES TO rdi_replication; +``` + +Repeat the schema grants for every schema you want RDI to capture. + +If RLS is enabled on a source table, the initial snapshot only contains rows +visible to the RDI role. To capture all rows, define appropriate RLS policies +for the role or grant `BYPASSRLS`: + +```sql +ALTER ROLE rdi_replication BYPASSRLS; +``` + +`BYPASSRLS` applies to every table in the database. Grant it only to a +dedicated RDI role and protect that role's credentials. + +### Create a publication + +By default, RDI uses the PostgreSQL `pgoutput` logical decoding plug-in, a +publication named `dbz_publication`, and a replication slot named `debezium`. +These defaults work with Supabase if the RDI role has permission to create the +publication and manage its source tables. + +It is recommended that a database administrator create a publication +containing only the tables RDI should capture: + +```sql +CREATE PUBLICATION rdi_publication + FOR TABLE public.customers, public.orders; +``` + +Creating the publication explicitly avoids granting table ownership or broad +publication-creation permissions to the RDI role and limits the publication's +table scope. + +## 5. Configure TLS + +In the Supabase dashboard, go to +[**Database settings** > **SSL configuration**](https://supabase.com/docs/guides/platform/ssl-enforcement): + +1. Enable **Enforce SSL on incoming connections**. +1. Download the Supabase CA certificate. + +Store the database username, password, and CA certificate as RDI secrets: + +```bash +redis-di set-secret SOURCE_DB_USERNAME rdi_replication +redis-di set-secret SOURCE_DB_PASSWORD '' +redis-di set-secret SOURCE_DB_CACERT /path/to/prod-ca-2021.crt +``` + +RDI verifies that the direct endpoint hostname matches the certificate. + +## 6. Configure RDI + +Add a PostgreSQL source to `config.yaml`. Replace the project reference and +table names with your values: + +```yaml +sources: + supabase: + type: cdc + connection: + type: postgresql + host: db..supabase.co + port: 5432 + database: postgres + user: ${SOURCE_DB_USERNAME} + password: ${SOURCE_DB_PASSWORD} + schemas: + - public + tables: + public.customers: {} + public.orders: {} + advanced: + source: + plugin.name: pgoutput + publication.name: rdi_publication + publication.autocreate.mode: disabled + slot.name: rdi_supabase +``` + +Use a unique replication slot name for each active pipeline that connects to +the project. + +## 7. Monitor replication slots + +RDI creates a logical replication slot that retains write-ahead log (WAL) +records while the pipeline is stopped or disconnected. Use a query like the +following to monitor inactive slots and retained WAL to prevent unexpected +storage growth: + +```sql +SELECT + slot_name, + active, + restart_lsn, + confirmed_flush_lsn +FROM pg_replication_slots; +``` + +[Supabase requires logical replication slots to be removed](https://supabase.com/docs/guides/platform/upgrading) +before a PostgreSQL major-version upgrade. Before upgrading: + +1. Stop the RDI pipeline. +1. Record the pipeline configuration and slot name. +1. Drop the RDI replication slot. +1. Upgrade the Supabase project. +1. Reset and start the RDI pipeline to create a new slot and initial snapshot. + +Allow time for the new initial snapshot to complete, and monitor the pipeline +until pending records return to zero. diff --git a/content/operate/rc/rdi/_index.md b/content/operate/rc/rdi/_index.md index bce780be1d..18064c5b6a 100644 --- a/content/operate/rc/rdi/_index.md +++ b/content/operate/rc/rdi/_index.md @@ -116,6 +116,7 @@ Before you can create a data pipeline, you must have: | MariaDB | 10.5, 11.4.3 | 10.4 to 10.11, 11.4.3 | | MySQL | 5.7, 8.0.x, 8.2 | 8.0.x | | PostgreSQL | 10, 11, 12, 13, 14, 15, 16 | 11, 12, 13, 14, 15, 16 | +| Supabase (uses PostgreSQL) | 10, 11, 12, 13, 14, 15, 16, 17 | - | | AWS Aurora PostgreSQL | 15 | 15 | | SQL Server | 2017, 2019, 2022 | 2016, 2017, 2019, 2022 | | MongoDB | 6.0, 7.0, 8.0 | - | diff --git a/content/operate/rc/rdi/setup.md b/content/operate/rc/rdi/setup.md index 85105ebc5a..88b34ac3fc 100644 --- a/content/operate/rc/rdi/setup.md +++ b/content/operate/rc/rdi/setup.md @@ -20,6 +20,7 @@ Before using the pipeline, you must first prepare your source database to use th See [Prepare source databases]({{}}) to find steps for your database type: - [MongoDB Atlas]({{}}) - [Snowflake]({{}}) +- [Supabase]({{}}) - Hosted on an AWS EC2 instance: - [MySQL and mariaDB]({{}}) - [Oracle]({{}}) @@ -494,4 +495,4 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv After you have set up your source database and prepared connectivity and credentials, select **Define source database** to [define your source connection and data pipeline]({{}}). -{{The define source database button.}} \ No newline at end of file +{{The define source database button.}} diff --git a/content/operate/rc/rdi/supabase.md b/content/operate/rc/rdi/supabase.md new file mode 100644 index 0000000000..be601fc39b --- /dev/null +++ b/content/operate/rc/rdi/supabase.md @@ -0,0 +1,129 @@ +--- +Title: Use Supabase with RDI +linkTitle: Supabase +alwaysopen: false +categories: +- docs +- operate +- rc +description: Connect a hosted Supabase database to RDI on Redis Cloud. +hideListLinks: true +weight: 7 +--- + +You can use a hosted [Supabase](https://supabase.com/) PostgreSQL database as +the source for an RDI pipeline on Redis Cloud. This page describes the +Redis Cloud-specific networking, secrets, and source configuration. + +The integration was validated with RDI 1.19.0 and hosted Supabase PostgreSQL +17.6. See [RDI on Redis Cloud prerequisites]({{< relref "/operate/rc/rdi#prerequisites" >}}) +for supported source versions. + +{{< warning >}} +Supabase AWS PrivateLink connectivity isn't supported. Supabase shares a +Resource Configuration through AWS Resource Access Manager and requires a +Resource-type VPC endpoint. This differs from the AWS PrivateLink +endpoint-service connectivity supported by RDI on Redis Cloud. + +Use the Supabase public direct database endpoint. +{{< /warning >}} + +## Before you begin + +You need: + +- A Redis Cloud RDI workspace and target database. +- A hosted Supabase project. +- A dedicated Supabase role with PostgreSQL replication and table-read + permissions. +- The Supabase CA certificate. +- An AWS Secrets Manager credentials secret and CA certificate secret shared + with Redis Cloud. + +Follow the steps in [Prepare Supabase for RDI]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase" >}}) +to create the database role, grant table access, account for Row Level +Security, and create a publication. + +## Configure public connectivity + +Supabase logical replication requires the direct database endpoint. Don't use +a Supavisor transaction or session pooler endpoint. + +The direct endpoint uses IPv6 by default, but RDI on Redis Cloud requires an +IPv4 endpoint, so you must enable the Supabase +[dedicated IPv4 add-on](https://supabase.com/docs/guides/platform/ipv4-address) +(you need a paid Supabase plan to do this). + +When you create the RDI pipeline: + +1. Select **PostgreSQL** as the source type. +1. Select **Public endpoint**. +1. Copy every **Redis Cloud outbound IP address** displayed by the setup flow. +1. In Supabase, open **Database settings** > **Network restrictions**. +1. Add every Redis Cloud outbound address as a `/32` CIDR. + +If you recreate the RDI workspace, its outbound addresses can change. Add the +new addresses to Supabase before starting the replacement pipeline, and remove +the old addresses after the new connection succeeds. + +## Configure secrets + +Follow the steps in [Share source database credentials]({{< relref "/operate/rc/rdi/setup#share-source-database-credentials" >}}) +to create and share: + +- A credentials secret containing the dedicated Supabase `username` and + `password`. +- A plaintext CA certificate secret containing the certificate downloaded + from Supabase **Database settings** > **SSL configuration**. + +Encrypt both secrets with the customer-managed AWS KMS key configured for the +RDI workspace. Use the AWS region that contains your Redis Cloud subscription. + +In the pipeline's **Secrets** section: + +1. Enter the credentials secret ARN. +1. Select **TLS** under **Transit security**. +1. Enter the CA certificate secret ARN. +1. Select **Validate**. + +RDI on Redis Cloud uses TLS and validates the Supabase CA certificate. + +## Configure the source + +In the RDI pipeline setup flow, open the source configuration and enter the +following values: + +| Field | Value | +|:--|:--| +| Source IP address / Hostname | `db..supabase.co` | +| Port | `5432` | +| Database | `postgres` | + +Under **Collector properties**, set: + +| Property | Value | +|:--|:--| +| `plugin.name` | `pgoutput` | +| `publication.name` | `rdi_publication` | +| `publication.autocreate.mode` | `disabled` | +| `slot.name` | A unique value, such as `rdi_supabase` | + +The publication name must match the publication you created in Supabase. Use a +unique replication slot name for each active pipeline connected to the +project. + +Select **Test source**. After the test succeeds, select the schemas and tables +to capture and deploy the pipeline. + +## Monitor the pipeline + +After deployment: + +1. Confirm the initial snapshot reaches zero pending and rejected records. +1. Insert, update, and delete test records in Supabase. +1. Confirm the corresponding counters increase in the pipeline metrics. + +Supabase logical replication slots retain write-ahead log (WAL) while the +pipeline is stopped. Follow the steps in +[Monitor replication slots]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase#7-monitor-replication-slots" >}}) +to monitor retained WAL and prepare for Supabase PostgreSQL upgrades.