Executable index surfaces over ExecutionWorkspace - #17
Merged
Conversation
nnunley
force-pushed
the
executable-index
branch
3 times, most recently
from
June 8, 2026 02:25
e1a1f55 to
40e0de8
Compare
… (STORY-0096) ITER-0001 dependency hygiene per the usage-site rule: the leit_wind_tunnel harness uses only rapidhash in its library surface; leit_core/leit_index/ leit_text are used solely by its #[cfg(test)] integration tests, so they move to [dev-dependencies] and no longer appear in the harness's production dependency graph. The bench crates were already correct (empty lib; all deps dev). Library build, 17 unit tests, and both bench crates verified green.
… (STORY-0112) ITER-0001: BlockId, FilterExprId, SegmentOrd, SegmentLocalDocId in leit_core, each a #[repr(transparent)] newtype over a [u8; 4] little-endian inner deriving bytemuck Pod/Zeroable. The on-disk form is the in-memory form: a &[u8] slice from an mmap'd buffer casts in place to &[Id] with no allocation or deserialization (zero-copy), stable across host endianness; ordering is numeric. bytemuck chosen over zerocopy because zerocopy's derives emit internal #[allow(non_ascii_idents)]/#[allow(non_local_definitions)] that conflict with the workspace's forbid-level Linebender lints (E0453); bytemuck is no_std and lint-clean under the same forbid set. Proven by SCENARIO-0005 (6 unit tests: value + slice + unaligned round-trip, numeric ordering, LE byte layout).
Records the design-decidable decisions for the Phase 2 segment format (DEC-01..10) with rationale, a Phase 3 forward-compatibility audit, and decision->enforcement traceability. Human-confirmed key calls: - DEC-01 segment offsets: u64 (no size cap; removes the only Phase 3 format-migration risk) - DEC-10 integrity: single footer checksum, verified in Full validation mode - DEC-06 block-aware API: public dedicated BlockCursor trait (Phase 3 WAND consumes it without a format/API break) - DEC-05 header: fixed-layout little-endian POD, absolute u64 section offsets, magic + version + format_flags, reserved stored-fields/columnar slots Decision-documentation ACs of STORY-0078/0081-0084/0090/0043-0047 are satisfied here (decided:ITER-0001); their code-enforcement ACs are deferred to ITER-0003/0004. Forward constraint recorded for ITER-0005: block-metadata schema must carry per-block max_score + doc-range for Phase 3 WAND/MaxScore.
…ORY-0112 AC-2) ITER-0001 audit corrective: SCENARIO-0005 now also exercises try_from_bytes/ try_cast_slice (Ok on well-formed, Err on malformed) per AC-2's validated-read obligation.
…elta) [ITER-0002] Codec layer for ITER-0002. A Codec trait with two implementations over a stable v1 block format, plus the layout decisions (DEC-11 fixed 128-doc blocks, DEC-12 layout) and a new TermFreq segment-resident type. - DeltaVarint (CodecId 0) + BlockDelta (CodecId 1, 128-doc independently-decodable blocks with validated first/last-doc header range). - Hand-rolled LEB128 varint into a type-enforced [u8;5]; no_std + alloc; no new deps. - API speaks named segment-resident types SegmentLocalDocId + TermFreq (no anonymous u32 drift); EntityId stays the in-memory abstraction, lowered at the segment boundary. - Decode into caller-provided &mut Vec<..> — scratch-ownership-agnostic (TODO(ITER-0003) / STORY-0079). Doc-sorted precondition enforced via checked_sub (deterministic panic). - CodecId marker per list; segment-format reservation deferred:ITER-0004. Stories: STORY-0002/0003/0004/0005(AC1-2)/0009 done; STORY-0087/0088 decided. Proof: SCENARIO-0006 (36 leit_postings tests). PAR spec + quality reviewed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ER-0002 T6-T7] SCENARIO-0070 (process-level): Criterion benchmark comparing DeltaVarint vs BlockDelta over the deterministic wind-tunnel corpus (1K/10K, multi-field, Zipfian). Measures encode time, decode time, and compressed size vs the 8-byte/posting baseline, with a lossless sanity gate. Baseline: DeltaVarint ~25%, BlockDelta ~26-27% of uncompressed; DeltaVarint decode ~4-11% faster. - crates/leit_wind_tunnel_index/benches/codec_compare.rs (+ [[bench]], leit_postings/ leit_core dev-deps). Criterion stays out of all primary crates (SCENARIO-0061/0069 pass). - leit_index: PostingEntry made public + InMemoryIndex::postings_by_term() accessor, the minimal surface needed to extract doc-sorted (SegmentLocalDocId, TermFreq) postings (lowering stands in for the ITER-0004 segment-write boundary). - docs/2026-05-30-codec-tradeoffs.md — STORY-0006 AC-3 decode-cost vs memory guidance. Stories: STORY-0006 (benchmark + guidance). Proof: SCENARIO-0070. PAR reviewed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ons [ITER-0003 T1] DEC-13: workspace-borrowed DecodeScratch (resolves TODO at codec.rs:153). DEC-14: layered cursor capabilities (DocCursor/TfCursor/BlockCursor; positions deferred). STORY-0079/0080/0089 decision ACs.
leit_postings cursor module over the ITER-0002 codec: - CursorStatus, DecodeScratch (workspace-borrowed, DEC-13), BlockSummary, PostingsView (borrowed view; decoupled from segment BlockMeta, TODO(ITER-0005) for sidecar lowering) - Layered DocCursor(current_doc/advance/advance_to)/TfCursor/BlockCursor (DEC-14) - CompressedCursor over decode_any; advance_to via binary search, never backward - CursorFactory (codec-agnostic) + BlockDecoder (real per-block decode for BlockDelta) - ScoreBound; block-aware API surface only (STORY-0024 AC-2 execution deferred Phase3; STORY-0081 AC-2 boundary) - 11 cursor tests (SCENARIO-0034/0035/0036/0038/0039/0041/0042/0043 + block API contract 0002/0009) - clippy clean (host + no_std aarch64-unknown-none), fmt clean
…iliation [ITER-0003 T7-T8] T7: allocation-evidence tests — scratch_reuse_no_realloc (SCENARIO-0039/0019/0024), hot_path_capacity_stable_and_cursor_small (SCENARIO-0040), and the empty-summaries conservative-default contract test (PAR B finding). T8: forward-extensibility module doc (STORY-0010); reconciled the legacy generic in-memory cursor traits with the canonical cursor:: API — InMemoryCursor methods are now inherent, the three colliding generic traits removed, canonical cursor traits re-exported at the crate root. Updated property_invariants.rs + phase1_readiness.rs consumers. clippy clean (host + no_std); workspace tests green.
…rsor [ITER-0003B T1] Add leit_index -> leit_postings dependency, record DEC-15 (dual cursor sources behind one trait-based executor: zero-copy in-memory cursor for the default path, CompressedCursor for the equivalence proof), and implement MemPostingsCursor — a zero-copy DocCursor+TfCursor over &[PostingEntry], semantically interchangeable with CompressedCursor. 11 cursor unit tests.
…0003B T2] Refactor InMemoryIndex term scoring into a single TfCursor-generic helper (score_via_cursor) driven by MemPostingsCursor on the default in-memory path, replacing direct Vec<PostingEntry> iteration in eval_term and collect_term. Block-level pruning, scoring math, and execution stats are byte-identical; the helper is generic over TfCursor so T3 reuses it with CompressedCursor. All leit_index + leit_integration_tests stay green (non-regression).
…6 [ITER-0003B T3] Thread a CursorSource selector through evaluate_plan_with_source -> evaluate_node_with_source -> eval_term_with_source so the compressed path (PostingsView + DecodeScratch + CompressedCursor over DeltaVarint/BlockDelta encoded postings) is a real execution path; existing callers default to InMemory (non-regression). SCENARIO-0026 integration test proves byte-identical ordered top-k across in-memory, DeltaVarint, and BlockDelta sources over single-term/AND/fielded queries, incl. a >128-posting term (multi-block decode). Closes STORY-0008 umbrella + STORY-0088 AC-2.
…TER-0003B T4] Add SCENARIO-0010 (STORY-0005 AC-3): a BlockDecoder::decode_block test proving partial postings retrieval decodes only the target block of a multi-block BlockDelta list (not the full list, no docs from earlier blocks). Lazy cursor-driven skip-during-decode is Phase 3 (CompressedCursor decodes eagerly). Iteration wrap-up: SCENARIO-0001/0015 mapped to the SCENARIO-0026 harness, STORY-0001 AC-2 to the existing BlockCursor API; EPIC-001 19/19.
Document the rationale for introducing a new fixed-header v1 segment format and for deprecating (rather than deleting) the original directory-based format, keeping the old reader as a frozen, backward-compatible shim that a future release removes.
Introduce a self-describing on-disk segment format alongside the existing in-memory index: - Fixed 80-byte header (little-endian): magic, version, format flags, and absolute u64 section offsets, with clean rejection of unknown versions and bad magic. - Trailing footer with a CRC32C checksum over the segment body for corruption detection. - Borrowed, zero-copy section readers (field table, lexicon with a term-bytes blob + fixed-width index, postings table, postings data) with bounds-checked, panic-free access on malformed input. - A segment writer that serialises an in-memory index into the format. - A borrowed SegmentView with three open-time validation modes (header-only, structural, full), trading startup cost for integrity guarantees. - Structured, no-heap-allocation segment error types. no_std + alloc compatible.
Add the fixed-width 12-byte block-metadata table entry — inclusive end document, a scorer-agnostic impact upper bound, and a payload-relative decode offset — as the v1 sidecar schema, with a zero-copy little-endian round-trip. Record the sidecar's physical placement, implicit per-term doc-range, and content decisions in the architecture notes.
Read per-block summaries (inclusive end document, impact upper bound, relative decode offset) from the block-metadata sidecar via alignment-free little-endian decoding, so the section may begin at any buffer offset. Empty, truncated, malformed, and out-of-range sections all surface a structured error instead of panicking, and reading summaries never touches the postings payload.
…writing Split each term's postings into fixed-size document windows and emit a per-block summary (inclusive end document, impact upper bound, and a payload-relative decode offset) into the block-metadata section. The postings-table entry now carries each term's first-block index and block count so a reader can locate its summaries, and the segment checksum covers the populated section. Block summaries let a reader skip to the relevant block without scanning the full postings payload.
Add the conversion from a segment block-metadata entry to the cursor-layer block summary, dropping the segment-only decode offset, so block-aware traversal can consume persisted summaries. Add a structural-overhead check asserting the sidecar costs a fixed twelve bytes per block and is readable without touching the postings payload.
…ecoded Assert the segment writer records the columnar offset in the header but writes no columnar bytes — the slot shares its offset with the stored-fields and footer positions — and confirm no reader decodes columnar content.
Promote the segment section accessors to the public surface, add field and term count accessors alongside the existing document count, and re-export the section reader types so a deserialized segment can be inspected through stable public API. A public round-trip test reads indexed field, term, and postings content back byte-for-byte using only the public surface.
Add an optional, std-only memory-mapping path: a segment can be opened from a file and borrowed as a zero-copy view whose lifetime is tied to the map handle. Header validation runs on open; structural and checksum validation run when a view is taken, so a valid-header/corrupt-body file is caught at view time. The mapped view produces byte-identical field, term, postings, and block-summary content to a buffer-backed view. The mapping is gated behind a cargo feature so the default no_std build is unaffected, and the segment thread-safety model is recorded in the architecture notes.
Add a version-dispatched migration entry point that validates a segment and returns it at the current format version — today an identity pass-through for the sole version, with an explicit dispatch seam for future conversions — and cleanly rejects unknown versions with an informative error rather than silently passing them through. Record the version support and deprecation policy in the architecture notes.
Add a two-multi-block-term segment whose terms cover disjoint document ranges and assert each term's block end-documents come from its own postings, never continued from the previous term's last block — pinning the implicit per-term doc-range derivation as a behavior, not just documentation.
nnunley
force-pushed
the
executable-index
branch
from
July 15, 2026 12:22
40e0de8 to
cdc5f16
Compare
nnunley
force-pushed
the
executable-index
branch
from
July 15, 2026 12:37
cdc5f16 to
d15b67c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Notes
Testing