Not blocking. Follow-up from the review rounds on the first PR.
The gap
The one rule proves a gate can fail. It says nothing about whether the gate is looking at the right thing, and nothing at all about the paths a fixture cannot reach.
Three classes of defect have now been found by review that the fixture model structurally cannot hold:
-
Anything reached only when a subprocess errors. grep exiting 2 on an unreadable directory, git ls-files exiting 128 on a corrupt index. Git does not store a mode-000 directory, and the same tree behaves differently depending on which uid CI runs as, so no committed tree reproduces either. Both were real false greens in the pinning gate and in nextjs-env; both were fixed and verified by hand (under an unprivileged uid, and against a deliberately corrupted index) rather than by a fixture.
-
False reds. The fixture model holds bad trees only, so a regression that rejects a valid form cannot be planted in one. Three shipped: a quoted uses: value, docker://image@sha256:<digest>, and docker:// appearing in a trailing comment. The tree leg catches these only if the valid form happens to appear in this repo's own tree.
-
Right verdict, wrong evidence. The errored-search branch used to leave the previous search's output in the hits file, so the denylist check printed a NEXT_PUBLIC_ line as its evidence. Exit code correct, output wrong. No exit-status-based test sees this.
fixtures/<gate>/cases/ narrows class 3's neighbours — each case is one tree holding one shape, so it can only pass by that shape still being caught — but it is still exit-code-only, and still bad-trees-only.
What would close it
Shape tables per parser: a list of (input line, expected verdict) pairs run directly against the matching function, with valid forms in the table as first-class entries. The three parsers that have each produced more than one defect:
- the
uses: matcher and value extractor in gates/actions-sha-pinned.sh — spaced key, quoted key, quoted value, inline comment, ./ local action, ./ inside a comment, docker:// by tag, docker:// by digest, docker:// in a comment, value on the following line, 40-hex pin, tag pin.
- the SQL comment stripper in
gates/migrations-lint.sh — line comments, block comments, a block comment opening and closing on one line, a block comment spanning statements, -- and /* inside a string literal. Its predecessor deleted from a same-line /* */ to EOF and two tables with no RLS reported clean.
in_git_repo and the tree-shape assumptions in gates/lib.sh — classic checkout, linked worktree, submodule, plain directory. .git being a file rather than a directory once skipped every git-backed check in two gates.
Two mechanisms worth considering for the error paths in class 1, both cheap:
- a
PATH shim that makes grep or git exit non-zero, asserting the gate reports rather than passes;
- running one selftest leg under an unprivileged uid so a mode-000 directory actually denies.
Assertions should cover stdout, not only the exit status — that is what separates class 3 from a pass.
Not in scope here
Extending check_references.py's recognised top-level directories. It is vendored byte-for-byte from vibeOS and a divergent second copy is the parallel-copies failure. That change belongs upstream.
Not blocking. Follow-up from the review rounds on the first PR.
The gap
The one rule proves a gate can fail. It says nothing about whether the gate is looking at the right thing, and nothing at all about the paths a fixture cannot reach.
Three classes of defect have now been found by review that the fixture model structurally cannot hold:
Anything reached only when a subprocess errors.
grepexiting 2 on an unreadable directory,git ls-filesexiting 128 on a corrupt index. Git does not store a mode-000 directory, and the same tree behaves differently depending on which uid CI runs as, so no committed tree reproduces either. Both were real false greens in the pinning gate and in nextjs-env; both were fixed and verified by hand (under an unprivileged uid, and against a deliberately corrupted index) rather than by a fixture.False reds. The fixture model holds bad trees only, so a regression that rejects a valid form cannot be planted in one. Three shipped: a quoted
uses:value,docker://image@sha256:<digest>, anddocker://appearing in a trailing comment. The tree leg catches these only if the valid form happens to appear in this repo's own tree.Right verdict, wrong evidence. The errored-search branch used to leave the previous search's output in the hits file, so the denylist check printed a
NEXT_PUBLIC_line as its evidence. Exit code correct, output wrong. No exit-status-based test sees this.fixtures/<gate>/cases/narrows class 3's neighbours — each case is one tree holding one shape, so it can only pass by that shape still being caught — but it is still exit-code-only, and still bad-trees-only.What would close it
Shape tables per parser: a list of
(input line, expected verdict)pairs run directly against the matching function, with valid forms in the table as first-class entries. The three parsers that have each produced more than one defect:uses:matcher and value extractor ingates/actions-sha-pinned.sh— spaced key, quoted key, quoted value, inline comment,./local action,./inside a comment,docker://by tag,docker://by digest,docker://in a comment, value on the following line, 40-hex pin, tag pin.gates/migrations-lint.sh— line comments, block comments, a block comment opening and closing on one line, a block comment spanning statements,--and/*inside a string literal. Its predecessor deleted from a same-line/* */to EOF and two tables with no RLS reported clean.in_git_repoand the tree-shape assumptions ingates/lib.sh— classic checkout, linked worktree, submodule, plain directory..gitbeing a file rather than a directory once skipped every git-backed check in two gates.Two mechanisms worth considering for the error paths in class 1, both cheap:
PATHshim that makesgreporgitexit non-zero, asserting the gate reports rather than passes;Assertions should cover stdout, not only the exit status — that is what separates class 3 from a pass.
Not in scope here
Extending
check_references.py's recognised top-level directories. It is vendored byte-for-byte from vibeOS and a divergent second copy is the parallel-copies failure. That change belongs upstream.