Running end-to-end-writer against dev, where compaction is enabled:
append.real_parquet_typed ops=40 wall_s=388 p50=9689ms rows_s=10314 rows=4000000
INVARIANT VIOLATION in end-to-end-writer: file_count=38, appended 41 files
The benchmark corrupted or mis-modeled catalog state; the numbers above are not trustworthy.
Nothing is corrupted. Compaction merged some of the appended files while the run was in flight, so the table legitimately holds fewer files than the writer registered.
The assumption
bench/src/hoglake_bench/scenarios/end_to_end.py:141:
check(info.file_count == appended, f"file_count={info.file_count}, appended {appended} files")Exact equality is only true if nothing else touches the table for the duration of the run — which holds on a laptop stack with compaction off, and does not hold on any deployed environment. The scenario models a catalog it owns exclusively.
Notably the row invariant on the next line (assert_row_tiling) passed, which is the right split: compaction preserves rows and is entitled to change file count. The benchmark asserted the one property that is not its own.
Fix
Assert what the writer controls, and bound rather than equate what it does not:
- rows tile exactly (already there, keep);
- every file the writer registered arrived with
stats_state == "provided" (already there — though note this too reads the current file set, so a merged-away file's stats are no longer observable; the check should be scoped to files the run actually registered, or relaxed to "no file is in a failed state");
file_count <= appended, since only compaction removes files and nothing else adds them. A count above the append count would still be a real violation worth catching.
Also worth having the failure message distinguish "fewer files than appended, which compaction explains" from a genuine inconsistency, so the next person does not have to reason it out from first principles.
Wider point
This is likely not the only scenario carrying an exclusive-ownership assumption. commit-throughput, changefeed-scan and ddl-churn were written against a quiet local stack too; anything asserting an absolute file or snapshot count will misfire the same way once maintenance loops are running. Worth a sweep for equality assertions over catalog state that another actor is entitled to change.
Running
end-to-end-writeragainst dev, where compaction is enabled:Nothing is corrupted. Compaction merged some of the appended files while the run was in flight, so the table legitimately holds fewer files than the writer registered.
The assumption
bench/src/hoglake_bench/scenarios/end_to_end.py:141:Exact equality is only true if nothing else touches the table for the duration of the run — which holds on a laptop stack with compaction off, and does not hold on any deployed environment. The scenario models a catalog it owns exclusively.
Notably the row invariant on the next line (
assert_row_tiling) passed, which is the right split: compaction preserves rows and is entitled to change file count. The benchmark asserted the one property that is not its own.Fix
Assert what the writer controls, and bound rather than equate what it does not:
stats_state == "provided"(already there — though note this too reads the current file set, so a merged-away file's stats are no longer observable; the check should be scoped to files the run actually registered, or relaxed to "no file is in a failed state");file_count <= appended, since only compaction removes files and nothing else adds them. A count above the append count would still be a real violation worth catching.Also worth having the failure message distinguish "fewer files than appended, which compaction explains" from a genuine inconsistency, so the next person does not have to reason it out from first principles.
Wider point
This is likely not the only scenario carrying an exclusive-ownership assumption.
commit-throughput,changefeed-scanandddl-churnwere written against a quiet local stack too; anything asserting an absolute file or snapshot count will misfire the same way once maintenance loops are running. Worth a sweep for equality assertions over catalog state that another actor is entitled to change.