Skip to content

Fix CI test failures by properly configuring data pipeline - #56

Merged
nee1k merged 1 commit into
mainfrom
fix/ci-test-data-pipeline
Jan 25, 2026
Merged

Fix CI test failures by properly configuring data pipeline#56
nee1k merged 1 commit into
mainfrom
fix/ci-test-data-pipeline

Conversation

@nee1k

@nee1k nee1k commented Jan 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes the CI test failures where tests expected nodes in Neo4j but found 0.

Root Cause

  1. Oracle daemon was connecting to external Kafka broker (cknbroker.pods.icicleai.tapis.io:443) instead of the local CI broker (broker:29092)
  2. Kafka security protocol was hardcoded to SSL, but CI broker uses PLAINTEXT
  3. CI wasn't waiting for Kafka Connect connectors to be ready before running tests
  4. Insufficient wait time for data to flow through the pipeline (daemon → Kafka → Neo4j)

Changes

  • oracle_daemon.py: Add CKN_KAFKA_SECURITY_PROTOCOL env var (defaults to SSL for backward compatibility)
  • docker-compose.ci.yml: New CI-specific override file that configures:
    • Local Kafka broker (broker:29092)
    • PLAINTEXT security protocol
    • Connection to ckn-network
  • ci.yml:
    • Add step to wait for Kafka Connect and Neo4j sink connector
    • Use docker-compose override for CI environment
    • Increase wait time for data ingestion

Test plan

  • CI passes "Build CKN" step
  • CI passes "Test CKN Oracle Daemon plugin WITHOUT power_monitoring" step
  • All 6 node count tests pass (Experiment, EdgeDevice, Deployment, User, Model, RawImage)

Issues fixed:
- Oracle daemon was connecting to external Kafka broker instead of local CI broker
- Kafka security protocol was hardcoded to SSL, but CI uses PLAINTEXT
- CI wasn't waiting for Kafka Connect connectors to be ready
- Data pipeline needed more time for events to flow through

Changes:
- Add CKN_KAFKA_SECURITY_PROTOCOL env var to oracle_daemon.py (defaults to SSL)
- Create docker-compose.ci.yml override for CI-specific configuration
- Add CI step to wait for Kafka Connect and Neo4j sink connector
- Use docker-compose override to connect daemon to local broker
- Increase wait time for data ingestion (daemon -> Kafka -> Neo4j)
@nee1k
nee1k merged commit 8c2ae5f into main Jan 25, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant