Skip to content

CockroachDBEndToEndIT passes without verifying end-to-end delivery #2001

@viragtripathi

Description

@viragtripathi

CockroachDBEndToEndIT.shouldStartTaskAndProduceSourceRecords is meant to validate the full pipeline (CockroachDB changefeed → intermediate Kafka → connector consumer → Debezium SourceRecords), but it passes even when zero records flow: it hard-asserts only that the task starts, and if no records are received it logs a warning and returns successfully.

In CI it collects zero records, for two reasons:

  1. The test manually pre-creates a changefeed with topic_name=..., but the connector's reuse detection matches on topic_prefix=..., so the connector does not recognize it and tries to create its own changefeed.
  2. The connector is configured with cockroachdb.changefeed.sink.uri = kafka://localhost:<host-mapped-port>. CockroachDB runs in its own container, so localhost is the CockroachDB container, not the host. CREATE CHANGEFEED fails with unable to dial ... connection refused, the connector retries indefinitely, and no events are produced.

Fix

  • Let the connector own the changefeed and point its sink URI at the internal Kafka network alias (kafka://kafka:9092), which CockroachDB can reach on the shared Testcontainers network.
  • Point the connector's own consumer at the host-mapped Kafka port via cockroachdb.changefeed.kafka.bootstrap.servers, since the consumer runs in the test JVM.
  • Insert data after the connector starts so the changefeed captures it.
  • Require that create, update, and delete SourceRecords are received, instead of passing on zero.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status
    In progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions