Fix CI test failures by properly configuring data pipeline - #56
Merged
Conversation
Issues fixed: - Oracle daemon was connecting to external Kafka broker instead of local CI broker - Kafka security protocol was hardcoded to SSL, but CI uses PLAINTEXT - CI wasn't waiting for Kafka Connect connectors to be ready - Data pipeline needed more time for events to flow through Changes: - Add CKN_KAFKA_SECURITY_PROTOCOL env var to oracle_daemon.py (defaults to SSL) - Create docker-compose.ci.yml override for CI-specific configuration - Add CI step to wait for Kafka Connect and Neo4j sink connector - Use docker-compose override to connect daemon to local broker - Increase wait time for data ingestion (daemon -> Kafka -> Neo4j)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the CI test failures where tests expected nodes in Neo4j but found 0.
Root Cause
cknbroker.pods.icicleai.tapis.io:443) instead of the local CI broker (broker:29092)SSL, but CI broker usesPLAINTEXTChanges
CKN_KAFKA_SECURITY_PROTOCOLenv var (defaults toSSLfor backward compatibility)broker:29092)PLAINTEXTsecurity protocolckn-networkTest plan