SOLR-18300 : Update apache.kafka to V4 - #4610
Conversation
…he-_-solr into renovate/major-apache.kafka
| for (TopicPartition topicPartition : topicPartitions) { | ||
| if (topicPartition.topic().equals(topic)) { | ||
| long endOffset = consumer.position(topicPartition); | ||
| long committedOffset = consumer.committed(topicPartition).offset(); |
There was a problem hiding this comment.
https://github.com/apache/solr/actions/runs/28428066221/job/84235821648 , Simply updating the version was giving compilation error.
Evidences for supporting this change:
| apache-httpcomponents-httpcore = { module = "org.apache.httpcomponents:httpcore", version.ref = "apache-httpcomponents-httpcore" } | ||
| apache-httpcomponents-httpmime = { module = "org.apache.httpcomponents:httpmime", version.ref = "apache-httpcomponents-httpmime" } | ||
| apache-kafka-clients = { module = "org.apache.kafka:kafka-clients", version.ref = "apache-kafka" } | ||
| apache-kafka-kafka213 = { module = "org.apache.kafka:kafka_2.13", version.ref = "apache-kafka" } |
| // Global override for local integration testing | ||
| grant { | ||
| permission java.security.AllPermission; | ||
| }; | ||
|
|
There was a problem hiding this comment.
[pending-to-discuss] with reviewer on the right way here. This works, but this is surely not the right way.
This is required to resolve this error, which comes in the cross-dc-manager test which interacts with the kafka container which runs on docker daemon.
| swagger3-jaxrs2-jakarta = { module = "io.swagger.core.v3:swagger-jaxrs2-jakarta", version.ref = "swagger3" } | ||
| tdunning-tdigest = { module = "com.tdunning:t-digest", version.ref = "tdunning-tdigest" } | ||
| testcontainers = { module = "org.testcontainers:testcontainers", version.ref = "testcontainers" } | ||
| testcontainers-kafka = { module = "org.testcontainers:testcontainers-kafka", version.ref = "testcontainers" } |
There was a problem hiding this comment.
Since Kafka 4.x fully supports lightweight, fast KRaft startups, the cleanest modern approach is to spin up an ephemeral Docker container using Testcontainers. It completely isolates Kafka from the JVM running Solr. [Industry standards]
There was a problem hiding this comment.
Makes sense. We do this for Tika testing...
|
Just curious: what motivated the upgrade? |
I came across this thread: https://lists.apache.org/thread/fkb9dgtybbjz4f5jzk6qb6ysh18bt5n5 by @janhoy asking for help in getting upgrades done & I landed upon the PR: #3274 that was raised by solrbot initially some 1 year back & was still pending. I thought of taking it to closure. Since there were major changes that were required, I created Jira & started working on it. |
Having experience what happened when we fell behind on Jetty upgrades and the fact that Solr 9 is now stuck on a EOL Jetty and only can get support from commercial suppliers, I don't want that to happen to other dependencies like Kafka. We NEED to not fall behind again. |
4 participants
https://issues.apache.org/jira/browse/SOLR-18300
Description
This branch is checked-out from the branch which solrbot created.
This PR is a superset of #3274 (-> can be closed once this is merged)
There were few major & significant changes introduced in apache.kafka v4, which are being done in this PR alongside the normal version bumps.
Solution
We were using EmbeddedKafkaCluster which in V4 is no longer supported.
Went ahead with the TestContainer approach & updated all tests in cross-dc-manager module accordingly.
AI Usage Disclosure - Used Gemini Chat (3.5 Flash) - for the guidance & code snippets.
Tests
./gradlew check -x test-> ✅Checklist
Please review the following and check all that apply:
mainbranch../gradlew check.