[bulk] Raise diff's lookahead to 30,000 so an unmatched item collection fits (closes #356) - #359
Merged
Merged
Conversation
…on fits The bound from #351 was set at 10,000 items, which turned out to be tight in two directions at once. Too tight for a legitimate case (#356). Realigning across an item collection the other table does not have means peeking past all of it, because every item in it carries the same pk -- so the whole collection is buffered to learn one fact. Two tables identical apart from one 12,000-item collection ended the run with "too different to diff accurately", which is the opposite of the disjoint pair the bound exists for. Measured against the real diff_segment: 9,990 and 10,000 diffed correctly, 10,100 and 12,000 failed, and with the cap lifted 12,000 and 50,000 both diffed completely. Also tighter than the budget needs. The @n columns in the constant's comment are already per-worker totals across all 8 buffers (two streams on each of four concurrent tasks), so 10,000 peaked at 888 MB of the ~6 GB a G.1X leaves the Python side -- most of the budget unused. 30,000 sits between the @10,000 and @50,000 columns: the worst normal item shape peaks around 2.6 GB, under half the budget, and an exhausted search runs ~1.5 s. Not higher, because overshooting is the worse failure. 50,000 puts that shape at 4.3 GB of ~6 GB, and this buffer is Python-side while the client's memory watchdog only recognises the JVM's wording -- so exceeding the budget here does not produce "the job ran out of memory, try R.1X", it produces a bare stop with no reason. #358 covers fixing that, and the cases 30,000 still cannot reach: collections wider than the cap, and items whose bulk is in large values, which no count of items bounds. Four tests hard-coded numbers coupled to 10,000. Two of them broke outright -- page_count 10,000 at 3 items per page is exactly the new cap, so the segment ended before the bound was reached. The other two silently stopped testing their invariant: they claimed "well past the lookahead limit" using 24,000 and 12,000, both now under it. All four now derive from MAX_LOOKAHEAD_ITEMS, so the next change to it cannot quietly weaken them. The new test fails at 10,000 with the exact message from #356 and passes at 30,000.
hunterhacker
force-pushed
the
raise-diff-lookahead-to-30k
branch
from
September 2, 2026 06:51
f3cd3d2 to
8039df2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #356.
#351's 10,000-item bound was too tight. Realigning across an item collection the other table lacks means peeking past all of it (every item shares one pk), so a collection wider than the cap ends the run with "too different to diff accurately" on tables that are otherwise identical.
Measured against the real
diff_segment: 10,000 items diffs fine, 10,100 fails, 12,000 fails; with the cap lifted 12,000 and 50,000 both diff completely.30,000 sits between the existing
@10,000and@50,000columns in the constant's comment — worst normal item shape peaks ~2.6 GB of the ~6 GB Python budget, exhausted search ~1.5 s. Not higher: 50,000 hits 4.3 GB, and this buffer is Python-side while the memory watchdog only knows the JVM's wording, so overshooting yields a bare stop, not the "try R.1X" diagnosis. #358 covers that and the cases 30,000 still can't reach.Tests: new case — a 12,000-item unmatched collection diffs completely (fails at 10,000 with #356's exact message, passes at 30,000). Four existing tests hard-coded 10,000 (two broke, two silently stopped testing their invariant); all now derive from
MAX_LOOKAHEAD_ITEMS.make test: 1721 passed, 48 skipped. Badges untouched.Server-side: needs
./bulk bootstrapto affect a live run.