eventservice: split large scanned transactions#5511
Conversation
|
Skipping CI for Draft Pull Request. |
ti-chi-bot
Bot
added
do-not-merge/needs-linked-issue
release-note
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds row-level scan resumption and spill-backed large-transaction processing across eventstore and eventservice. It also adds progress metrics, cleanup paths, integration coverage, a dispatcher-reset failpoint, and a dependency declaration update. ChangesLarge Transaction Split & Spill
Dispatcher batch-reset failpoint
Dependency declaration
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant EventBroker
participant DispatcherStat
participant EventScanner
participant EventStore
participant SpillFile
EventBroker->>DispatcherStat: request scan cursor
DispatcherStat-->>EventBroker: range and resume position
EventBroker->>EventScanner: scan request
EventScanner->>EventStore: iterate with cursor
EventStore-->>EventScanner: event and scan position
EventScanner->>SpillFile: append large-transaction inserts
EventScanner-->>EventBroker: events and scan progress
EventBroker->>DispatcherStat: persist scan progress
Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ti-chi-bot
Bot
added
the
size/XXL
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for row-level resume tokens and large transaction spilling in the event service. It adds a new EventIteratorWithScanPosition interface, updates the event store iterator to track and return opaque scan positions, and implements disk-based spilling for large transactions via largeTxnInsertSpill and largeTxnScanState. Review feedback identifies two key issues in large_txn_spill.go: a potential resource leak where a failed Close() in Cleanup() skips deleting the temporary file, and a loose record length check that could lead to OOM panics if the spill file is corrupted.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Signed-off-by: dongmen <414110582@qq.com>
# Conflicts: # pkg/eventservice/event_scanner.go
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
Introduce pkg/spill as a shared temp-file record store with chunked append, handle reads, sequential reads, and cleanup. Refactor eventservice large transaction insert spill to reuse it while keeping RawKVEntry encoding local.
ti-chi-bot
Bot
added
do-not-merge/needs-triage-completed
and removed
do-not-merge/needs-linked-issue
labels
|
One thing to note: may be we need respect CMEK for the spill file? |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
pkg/eventservice/large_txn_spill.go (1)
104-109: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winThread the scan context through spill encryption
AppendandNextpasscontext.Background()intoEncryptData/DecryptData; on cache miss those calls can block on TiKV metadata fetch or KMS decrypt, so the spill path ignores cancellation when the dispatcher is removed or the scan aborts. Pass the caller’s context instead.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/eventservice/large_txn_spill.go` around lines 104 - 109, Update the spill encryption flow in Append and Next to pass each method’s caller-provided context to EncryptData and DecryptData instead of context.Background(). Preserve the existing encryption/decryption and error propagation behavior while ensuring cancellation reaches metadata and KMS operations.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/eventservice/txn_scan_strategy.go`:
- Around line 453-456: Update getOrCreateLargeTxnState to return the appropriate
predefined repository error for a missing dispatcherStat instead of constructing
a new errors.New value. Reuse the existing error symbol used for this invariant,
following the repository’s established error-handling conventions and preserving
the current failure path.
---
Nitpick comments:
In `@pkg/eventservice/large_txn_spill.go`:
- Around line 104-109: Update the spill encryption flow in Append and Next to
pass each method’s caller-provided context to EncryptData and DecryptData
instead of context.Background(). Preserve the existing encryption/decryption and
error propagation behavior while ensuring cancellation reaches metadata and KMS
operations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: d38fda9a-5d90-40b8-bc12-e75838a11eab
📒 Files selected for processing (19)
logservice/eventstore/event_store.gologservice/eventstore/event_store_test.gologservice/eventstore/format.gologservice/eventstore/pebble_test.gologservice/eventstore/scan_request.gopkg/common/table_span.gopkg/eventservice/dispatcher_stat.gopkg/eventservice/dispatcher_stat_test.gopkg/eventservice/event_broker.gopkg/eventservice/event_broker_test.gopkg/eventservice/event_scanner.gopkg/eventservice/event_scanner_benchmark_test.gopkg/eventservice/event_scanner_test.gopkg/eventservice/event_service_test.gopkg/eventservice/large_txn_spill.gopkg/eventservice/large_txn_spill_test.gopkg/eventservice/large_txn_state.gopkg/eventservice/scan_progress.gopkg/eventservice/txn_scan_strategy.go
🚧 Files skipped from review as they are similar to previous changes (5)
- logservice/eventstore/format.go
- logservice/eventstore/pebble_test.go
- pkg/eventservice/large_txn_state.go
- pkg/eventservice/event_broker.go
- pkg/eventservice/event_scanner_test.go
Signed-off-by: dongmen <414110582@qq.com> # Conflicts: # pkg/eventservice/event_broker.go
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
|
/test all |
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
Document scan progress and transaction scan strategies, and simplify the spill insert drain control flow without changing its behavior. Signed-off-by: dongmen <414110582@qq.com>
asddongmen
added
the
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: lidezhu, wk989898 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
ti-chi-bot
Bot
added
lgtm
approved
and removed
needs-1-more-lgtm
Yes, thank you. I have added that part. |
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
Signed-off-by: dongmen <414110582@qq.com>
|
/test pull-cdc-mysql-integration-heavy-next-gen |
Signed-off-by: dongmen <414110582@qq.com>
|
/test pull-cdc-mysql-integration-heavy-next-gen |
|
@asddongmen: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
3 participants
What problem does this PR solve?
Issue Number: close #5596
EventBroker scans a whole large transaction, which can OOM CDC.
What is changed and how it works?
This PR adds row-level EventStore resume positions so
transaction-atomicity=noneemits bounded fragments without advancing resolved-ts early. Deferred insert halves of unique-key updates are spilled to disk and drained or cleaned across reset/remove.TiUP E2E comparison
Runs used one CDC limited to 32 GiB, upstream/downstream TiDB, and the same workload.
masterThe 20 GiB OOM is caused mainly by realtime logpuller
txnMatcherPREWRITE caching and overlapping ingestion buffers, before EventBroker scan splitting applies. asddongmen/tigate#2 fixes this matcher pressure and passed a separate 20 GiB no-OOM test; whether to merge it will be evaluated separately.Check List
Tests
The integration-test suite covers all changed workflows: row resume, interruption, unique-key spill/drain, CMEK spill encryption, dispatcher reset, and end-to-end consistency. Focused unit tests cover edge cases.
Questions
Will it cause performance regression or break compatibility?
No compatibility break is expected. Splitting applies only when transaction atomicity permits it; unique-key updates may add local spill I/O.
Do you need to update user documentation, design documentation or monitoring documentation?
An internal design document is included; no user-facing update is required.
Release note