Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR improves the offset auto reset logic in seekable stream supervisors by centralizing reset responsibility to supervisors only and improving error messaging. The changes ensure only supervisors handle auto resets while tasks always throw OffsetOutOfRangeException, and provide clearer error messages when offsets are reset.
- Enhanced supervisor-only auto reset logic with better offset availability checking
- Improved error messages for task shutdown during offset resets
- Removed task-level auto reset handling to centralize control in supervisors
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| SeekableStreamSupervisor.java | Modified resetInternal method to accept autoReset flag and improved offset availability checking logic |
| KafkaIndexTaskRunner.java | Removed task-level auto reset handling, now only throws OffsetOutOfRangeException |
| SeekableStreamSupervisorStateTest.java | Updated test expectations for new error message format and added recordSupplier mock |
| KinesisSupervisorTest.java | Updated test calls to resetInternal method and adjusted test expectations |
| KafkaSupervisorTest.java | Updated test calls to resetInternal method and corrected test data expectations |
.../main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
Show resolved
Hide resolved
.../main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
Show resolved
Hide resolved
.../main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
Outdated
Show resolved
Hide resolved
…blestream/supervisor/SeekableStreamSupervisor.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
This pull request has been marked as stale due to 60 days of inactivity. |
|
This pull request/issue has been closed due to lack of activity. If you think that |
Fixes #18282
Description
Improve the auto reset logic as proposed in the above issue:
OffsetOutOfRangeExceptionif this exception is raised, but will publish ingested message under such caseDataSourceMetadata is not found while resetin task report to make it intuitive when supervisor kills a task because of offset resetNOTE
this relies on #18226, which publishes segments when OffsetOutOfRangeException is raised.
This PR has: