fix(pipeline): make the #334 plausibility gate fire (capture committed_nodes before dump)#456
Open
nguiaSoren wants to merge 2 commits into
Open
fix(pipeline): make the #334 plausibility gate fire (capture committed_nodes before dump)#456nguiaSoren wants to merge 2 commits into
nguiaSoren wants to merge 2 commits into
Conversation
Compare persisted SQLite node counts to in-memory dump counts after index_repository completes so partial WAL/durability loss surfaces as status:"degraded" instead of silent indexed. Signed-off-by: Sam Li <yangsec888@gmail.com>
…eusData#334) The DeusData#334 plausibility gate compares committed (extracted) node count against persisted rows, but committed_nodes was read AFTER cbm_gbuf_dump_to_sqlite(), which calls release_gbuf_indexes() and frees node_by_qn. cbm_gbuf_node_count() therefore returned 0, committed_nodes fell at/below CBM_DUMP_VERIFY_MIN_FLOOR, and the gate never fired (a healthy index reported expected_nodes:0 while expected_edges was correct, since edges.count survives the dump). - pipeline.c: capture committed node/edge counts before the dump. - pipeline_incremental.c: set committed counts on the incremental path too (via new cbm_pipeline_set_committed_counts), so the gate covers incremental/reparse reindexes — the scenario DeusData#334 actually reproduces. - test_pipeline.c: drive the real pipeline and assert committed_nodes > 0 and == persisted (the existing dump_verify tests fed synthetic counts and could not catch the capture-ordering bug). Signed-off-by: soren <nguiasoren@gmail.com>
Author
|
Friendly ping on this — CI is green across all platforms (tests, lint, security, DCO) and @yangsec888 has signed off on the approach. Happy to rebase or adjust anything if it'd help it land. No rush, and thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on #433 (the
status:degradedplausibility gate by @yangsec888) and fixes a capture-ordering bug that left the gate inert — it never tripped, so a catastrophically partial index would still reportstatus: "indexed", the exact silent-success #334 set out to stop. @yangsec888 reviewed the finding and deferred to this fix (#433 (comment)).Closes #334. Supersedes #433 (its gate commit is included here with authorship preserved).
The bug
dump_and_persist_hashesread the committed node count atpipeline.caftercbm_gbuf_dump_to_sqlite(), which runsrelease_and_remap_vectors()→release_gbuf_indexes()and freesnode_by_qn(graph_buffer.c:348).cbm_gbuf_node_count()therefore returned0. Withcommitted_nodes = 0 ≤ CBM_DUMP_VERIFY_MIN_FLOOR,cbm_dump_verify_is_degraded()short-circuits tofalse.Edges survive the free (
edge_by_keyis freed, butedges.countis not), which is why a healthy full index reported a plausibleexpected_edgeswhileexpected_nodescame back0:Separately, the incremental path (
pipeline_incremental.c) never set the committed count at all, so the gate was also disabled on incremental/reparse reindexes — the scenario #334 actually reproduces (repeatedindex_repositoryon an already-present index).The fix
pipeline.c— capturecommitted_nodes/committed_edgesbefore the dump.pipeline_incremental.c+pipeline_internal.h— record committed counts on the incremental path via a newcbm_pipeline_set_committed_counts()(the incremental unit cannot see the opaquecbm_pipelinestruct).tests/test_pipeline.c—pipeline_committed_counts_match_persisted: drives the real pipeline and assertscommitted_nodes > 0and== persisted. The existingdump_verifytests feed the predicate synthetic values, so they cover the pure function but not the pipeline wiring; this test fails on the pre-fix code and passes after.Verification
scripts/build.sh).expected_nodes == nodeson both the full and incremental paths (was0before).scripts/test.sh→ 5,620 passing, including the new test.committed_nodes == 0) and passes after, so it genuinely guards the regression.All commits are DCO signed-off.