Add native CockroachDB connector support - #3861
Conversation
eb56599 to
31e5672
Compare
31e5672 to
f16c84e
Compare
f16c84e to
2f33f5d
Compare
|
Rebased onto current main and expanded this from a peer-only connector to a full source integration:
Tested against CockroachDB v25.4.13 (e2e suite, single node insecure) and a CockroachDB Cloud v26.2.1 cluster over verify-full TLS for connectivity, validation and schema introspection. A docs PR to peerdb-io/docs will follow once this merges. |
Implements CockroachDB as a source peer with: - Native Go connector using the pgx driver (peer creation, validation, version gating, variant detection, schema introspection) - SQL syntax support (CREATE PEER ... FROM COCKROACHDB) via the in-tree parser, with nexus query routing through the Postgres executor since CockroachDB is wire compatible - QRep replication with AS OF SYSTEM TIME partitioned reads and watermark based partitioning - CDC via sinkless changefeeds: resolved timestamp checkpoints, cursor resume with backoff, gapless snapshot handoff using cluster_logical_timestamp, delete key fallback, replay dedup, SQLSTATE based error classification - Complete UI integration including a changefeeds toggle - Unit tests, an e2e suite, and CockroachDB wired into CI Tested against CockroachDB v25.4.13 single node (insecure) with the e2e suite and against a CockroachDB Cloud v26.2.1 cluster (TLS verify-full) for connectivity, validation and schema introspection.
2f33f5d to
0fa9e5a
Compare
|
Updated: the rangefeed validation check queried crdb_internal.cluster_settings, which is gated behind allow_unsafe_internals from v26.1 onward, so on newer clusters the check silently returned nothing and mirror validation passed without verifying rangefeeds. Switched it (and the database variant lookup) to SHOW CLUSTER SETTING, which only needs VIEWCLUSTERSETTING, and added an e2e test covering the rangefeed validation path against a live cluster, including the disabled-setting error case. Planned follow-ups: changefeed sink transports (webhook and Kafka) for deployments that need changefeed job semantics, protected timestamps through large initial scans, and DB Console metrics. The changefeed envelope parsing and checkpointing in this PR are transport independent, so those land as additional transports rather than a redesign. |
Summary
Implements full CockroachDB integration as a source peer, following the same patterns as existing PostgreSQL and MySQL connectors.
What's included
Backend:
Frontend:
Testing performed
Design Decisions
TLS Default: Set to
false(matching Postgres) for easier local development. Cloud users will enable it as needed.SSH Support: Backend implementation included but no UI exposed. CockroachDB Cloud deployments (primary use case) don't require SSH tunnels, keeping the UI simpler and more focused.
Note on SQL syntax
SQL syntax support (
CREATE PEER FROM COCKROACHDB) requires updates to the sqlparser dependency and will be added in a follow-up PR.Files changed
23 files changed, 936 insertions(+), 2 deletions(-)