feat(goap): add the ADR-0095 scale-evidence runner, artifacts and measured memory model - #733
Merged
Merged
Conversation
added 4 commits
September 17, 2026 21:28
The two P2 evidence actions could never fire: both listed `performance_claims_have_current_artifacts: true` as a precondition, but no action's `effects:` set that flag — it was false in GOAP_STATE and GOALS, so the planner had a cycle with no entry point. - ACTIONS.md: `add_ann_and_persistence_scale_benchmarks` now *produces* the flag (it is the action that writes the scale artifacts), so `performance_claims_have_current_artifacts` moved from its preconditions to its effects. `replace_formula_only_memory_claim` keeps the precondition, which the first action now satisfies. - GOAP_STATE.md: added the missing `adr_0095_accepted: true` (ADR-0095 is Accepted 2026-07-16), the unsatisfiable precondition of `deduplicate_test_and_source_surfaces`. - GOALS.md: `workspace_implementation_owners_unique` said false with a "csm-cli/csm-wasm pending" comment while GOAP_STATE records true since 2026-08-12 (#626/#627); trued up.
Implements the measurement half of `add_ann_and_persistence_scale_benchmarks` and `replace_formula_only_memory_claim`. `examples/scale_evidence` is a three-mode runner (release-only, `required-features = persistence,ann-hnsw, ann-lsh`): - `ann`: exact (brute force), HNSW and LSH indexes measured directly through the `AnnIndex` trait plus the bucketed candidate path through `Singularity`; reports build time, query p50/p95/p99, relevance-based recall@k against the exact ground truth, index bytes, serialize bytes, reload time and delete/re-insert cost. - `persistence`: sequential save throughput and batch percentiles, the read path, and DB/WAL/SHM bytes reported separately; plus a concurrent-writer probe reported twice — raw (no retries) and with a caller-side bounded retry policy — because the library exposes no retry or timeout of its own. - `memory`: per-scale-point child processes measure RSS delta and persisted bytes with a streaming corpus generator (no second copy of the corpus in the measured process), fit `bytes = intercept + slope * concepts`, score the fit on a held-out point and project the configured claim size (12 MiB / 10M by default). `scripts/scale-evidence.sh` wraps a mode with the ADR-0095 evidence manifest: schema version, commit, dirty state, command, features, corpus version/seed/ checksum, rustc and cargo versions, OS/kernel/arch/CPU/cores/RAM, sample count, baseline note and the reported variance statistics.
A single evidence.json was overwritten by each mode, so only the last run's manifest survived. Name them evidence_<mode>.json instead.
Artifacts produced by `scripts/scale-evidence.sh` at commit 4d81491 (clean tree, release profile) — see plans/evidence/scale_2026_09_17/README.md for the rendered tables and manifests. What the measurements show: - ANN at 50 k: HNSW answers in 896 us p50 for 0.892 recall@10 (exact is 6.35 ms p50 for 1.000), but costs 80 s to build and 93 MB serialized; LSH answers in 813 us for 0.836 recall with a 282 ms build. Bucketed candidate generation (probe width 8) loses ~41 % of the true top-10 (0.590 recall). - Persistence: 5.2 k concepts/s sequential writes, 2.85 kB/concept on disk; under 8 concurrent writers 90 % of operations fail with no retry and 47 % still fail with a caller-side bounded retry — the library has no busy timeout or retry policy. - Memory: RSS 4 691 B/concept and storage 2 850 B/concept, held-out (100 k) error 0.29 % and 0.06 %. The 10 M projection is 43.7 GB RSS / 26.5 GB storage, so the recorded `< 12 MB` target is exceeded by 3 728x / 2 265x and `claim_supported` is false. The claim correction, the persistence retry/timeout change and the GOAP flag updates follow in separate commits.
Contributor
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 78 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
d-oit
changed the title
Opengrep flags std::env::args() and std::env::current_exe() as high-severity in examples/scale_evidence/main.rs. The harness uses args for measurement parameters and current_exe to re-execute itself for a clean-process RSS baseline - neither feeds a security decision. Excluded at file level in .codacy.yml, the same escape hatch used for the SIMD hot paths, rather than an un-reviewable dashboard suppression.
The directory path without a glob did not match, so opengrep still reported the two high-severity findings on the harness. Use examples/scale_evidence/** like the other path patterns in this config.
…odel ADR-0095 forbids memory claims built from constants. The recorded `< 12 MB for 10M concepts` target describes ADR-0024 phase-2 product quantization (1 byte/concept + 2 MB codebook), which was never implemented: the shipped representation stores a 1 280-byte `HVec10240` per concept plus index copies and writes it twice (concept row + version row). Measured on commit 4d81491 and published in `plans/evidence/scale_2026_09_17/memory_model.json`: RSS 4 691 B/concept and 2 850 B persisted/concept, held-out error 0.29 % / 0.06 % at 100 k concepts. The 10 M projection is therefore 43.7 GB RSS / 26.5 GB on disk. - `tests/performance_targets.rs`: `projected_10m_concepts_memory_stays_under_12mb` and its env-tunable constants are replaced by `measured_memory_footprint_matches_the_evidence_model`, which saves 10 000 concepts, sums db+wal+shm bytes after a checkpoint, asserts the measured bytes/concept band and prints the RSS delta and the 10 M extrapolation. - `book/src/performance.md` and `docs/architecture/context.yaml` mark the target not supported with the measured slope, evidence path and the attribution to the unimplemented design. - CHANGELOG records the correction under Unreleased.
d-oit
changed the title
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Delivers two queued ADR-0095 actions end to end:
add_ann_and_persistence_scale_benchmarks(measurement half) andreplace_formula_only_memory_claim(measured model + claim correction).Runner (
examples/scale_evidence,required-features = persistence,ann-hnsw,ann-lsh)Three modes, all releasing in the artifact writer:
ann— exact (brute force), HNSW and LSH indexed directly through theAnnIndextrait, plus the bucketed candidate path throughSingularity. Reports build time, query p50/p95/p99, relevance-based recall@k against the exact ground truth, index bytes, serialized bytes, reload time and delete/re-insert cost.persistence— sequential save throughput and batch percentiles, read path, DB/WAL/SHM bytes reported separately, and a concurrent-writer probe reported twice: raw (no retries) and with a caller-side bounded retry policy. The library has no retry/timeout of its own, so the harness has to supply one — reported as evidence rather than hidden.memory— one child process per scale point (clean RSS baseline) with a streaming corpus generator, so the measured process never holds a second copy of the corpus. Fitsbytes = intercept + slope × concepts, scores the fit on a held-out point, and projects the configured claim (12 MiB / 10M by default).scripts/scale-evidence.shwraps a mode with the ADR-0095 manifest: schema version, commit, dirty state, command, features, corpus version/seed/checksum, rustc/cargo versions, OS/kernel/arch/CPU/cores/RAM, sample count, baseline note, variance statistics. One manifest per mode (evidence_<mode>.json).Committed artifacts (
plans/evidence/scale_2026_09_17/)Produced at commit
4d81491, clean tree, release profile. Headline numbers:< 12 MBtarget (claim_supported: false)The 12 MB figure belongs to the product-quantization design (ADR-0024 phase 2) that was never implemented; the shipped representation stores a 1 280-byte
HVec10240per concept (plus index copies) and writes it twice (concept + version rows).Verification
cargo clippy --example scale_evidence --features ann-hnsw,ann-lshclean;cargo fmt --all -- --checkclean.plans/evidence/scale_2026_09_17/README.mdis rendered from the committed JSON, so numbers and prose cannot drift.Claim correction
The recorded
< 12 MB for 10M conceptstarget describes ADR-0024 phase-2 product quantization, which was never implemented. Replaced wherever it was asserted as passing:tests/performance_targets.rs: the formula test and its env-tunable constants are gone;measured_memory_footprint_matches_the_evidence_modelsaves 10 000 concepts, sums db+wal+shm after checkpoint, asserts the measured bytes/concept band and prints the RSS delta and 10 M extrapolation.book/src/performance.md,docs/architecture/context.yaml: target marked not supported with the measured slope, the evidence path and the attribution to the unimplemented design.CHANGELOG.md: correction recorded under Unreleased..codacy.yml: the harness is excluded from therust.lang.security.args/current-exeopengrep rules (measurement parameters and self re-exec for a clean RSS baseline, no security decision); verified withcodacy pull-request→ 0 new issues.Follow-ups (separate PRs)
csm-persistence; after this PR lands it re-runs the persistence artifact so the directory carries before and after numbers.performance_claims_have_current_artifacts/ann_scale_evidence_current/persistence_contention_evidence_current/measured_memory_model_exists/ten_million_memory_claim_evaluatedinplans/GOAP_STATE.mdand record the wave inprogress/.