logpuller: spill realtime large transaction prewrites#2
Draft
asddongmen wants to merge 2 commits into
Draft
Conversation
asddongmen
force-pushed
the
log-puller-big-txn-optimize
branch
5 times, most recently
from
July 7, 2026 05:17
b2a475f to
353d657
Compare
Avoid attaching cached prewrite Value and OldValue to COMMIT rows that are already stale for the subscribed span. Cached stale COMMIT rows are still matched and removed during initialization, but dropped without retaining large payloads.
Spill unmatched prewrite values once the matcher crosses the memory threshold. Flush matched KV event cache in bounded chunks during large COMMIT batches while preserving the final async wake path. Clear protobuf row value references after the matcher or KV cache takes ownership, and force GC after releasing large intermediate chunks. Apply backpressure from large dynstream entry events back to the TiKV receive loop so COMMIT value batches cannot accumulate in the puller buffer while the handler is waiting on eventstore writes.
asddongmen
force-pushed
the
log-puller-big-txn-optimize
branch
from
July 7, 2026 06:32
353d657 to
499ac35
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: ref pingcap#5561
This is stacked on
0626-split-big-txn. Realtime large transactions can retain large PREWRITEValue/OldValuepayloads in logpuller matcher before eventstore processing, and stale COMMIT rows could temporarily attach values that are dropped immediately.What is changed and how it works?
pkg/spillrecord-file utility from the base branch to spill unmatched prewrite payloads after a 1 MiB per-matcher in-memory threshold is exceeded.Check List
Tests
Manual test details
Before splitting the shared spill commit into the base branch, equivalent code at commit
353d65779was patched toroot@10.2.15.7and one 20GB transaction was run through changefeedlogpuller-thr-20g-131911.Result:
2097152020971520normal2026-07-07 13:43:37.141After the branch split,
log-puller-big-txn-optimizecontains only logpuller changes relative to0626-split-big-txn.Questions
Will it cause performance regression or break compatibility?
No compatibility change expected. The spill path trades disk IO for bounded matcher heap usage only when unmatched prewrite payloads exceed the per-matcher threshold.
Do you need to update user documentation, design documentation or monitoring documentation?
No.
Release note