Skip to content

Ported from spar's self-audit: 2 verification artifacts cite tests that never run (exit 0), and MSRV is ungated #911

Description

@avrabe

spar has been running a self-audit and filing what it finds (pulseengine/spar
#358, #369, #382#385, #388#391). The taxonomy is the same one synth kept
hitting in v0.54
— gates that pass without checking — so I tested synth
against each spar finding rather than assuming we differ. Three replicate.

1. spar#388 replicates — cargo test -- <no match> exits 0

Two rivet verification artifacts cite a test-name filter that exists nowhere
under crates/:

artifact cited command
artifacts/e2e-verification.yaml:363 cargo test -p synth-backend -- test_meld_import_table
artifacts/e2e-verification.yaml:387 cargo test -p synth-backend -- test_startup_memory_base

Proven, not inferred:

$ cargo test -p synth-cli -- test_meld_import_table
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 56 filtered out
→ exit 0

A filter that matches nothing is indistinguishable from a passing test. Both
artifacts therefore claim verification by evidence that never runs — the same
shape as the sret_decide_differential.py gate v0.54 found printing
MISMATCH <-- BUG and exiting 0.

Fix has two parts: correct or remove those two citations, and make the class
unrepresentable — a check that every run: in artifacts/*.yaml naming a test
filter resolves to a real test, and/or --exact/--no-tests=fail where the
runner supports it. Fixing only the two instances repeats the #890 mistake of
fixing output instead of the factory.

Scope measured while checking: 29 cargo test commands are cited across the
rivet artifacts; 13 use a -- name filter; 2 of those 13 resolve to
nothing. All --test <target> citations (the other form) resolve correctly.

2. spar#369 replicates — nothing gates the MSRV

Cargo.toml:33 declares rust-version = "1.88", and CLAUDE.md repeats it as a
project constraint. No workflow job builds at 1.88 — every CI job uses
dtolnay/rust-toolchain@stable. So the real minimum supported version drifts
upward silently on any dependency bump that raises its own MSRV, with no repo
diff and no red build. The declared number is currently an aspiration, not a
tested property.

Cheap fix: one cargo check --workspace --locked job pinned to the declared
rust-version, so the claim and the artifact move together.

3. spar#358 shape — the verification-scope question

spar found its verification gate checking 20 of 161 feature artifacts. synth has
255 rivet artifacts; I have not established how many have their cited
evidence actually executed in CI, and the answer is not obviously good given
finding 1 above. This is the rivet-artifact twin of #890 (which asked the same
question of scripts/repro/*.py and found 70 undeclared). Worth measuring
before claiming the trace is closed.

What I checked and found CLEAN

  • spar#383 (format check green on unformatted code — cargo fmt --all
    covering only part of the repo): synth declares 18 workspace members and
    has 18 crate directories, none missing. cargo fmt --all --check genuinely
    covers everything.
  • spar#380 (strict: true with no merge queue making concurrent CI dead
    work): N/A — synth sets strict: false deliberately, recorded when branch
    protection was configured.
  • The --test <target> form of artifact citations: 29 checked, 0 broken.

Why port these at all

The v0.54 release fixed nine defects that were in checkers rather than in the
code they check
. spar independently found the same class in its own tooling.
That two sibling projects converge on it suggests it is a property of
verification-heavy repos generally: once the artifact is well-tested, the
residual defects migrate into the instruments, where they are invisible because
the instrument's whole job is to report "fine".

Credit: pulseengine/spar#388 (no-match exit 0), #369 (MSRV closure drift),
#358 (verification gate scope).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions