fix(inspect): score on execution evidence — close review findings M2/M3 - #30
Merged
Merged
Conversation
…ot tokens The inspector's false-completion-defense credit was fully gameable: a keyword-stuffed fake scored 100/100. Two holes: - _gate_invoked_in_verify credited any verify-* line containing a gate token, so echo "holdout_gate" (and printf / ":" / quoted-string variants) earned full "invoked" credit without executing anything. - _gate_run_recorded credited a bare gate token anywhere in RUNLOG.md prose, so stuffed narration sufficed — and because a token like anticheat_scan contains the run-word "scan", even a token+run-word rule would self-satisfy. Now credit requires genuine execution evidence: the gate script (holdout_gate.py / anticheat_scan.py / anti_cheat.py) invoked as a command (not printed by an inert emitter), or a RUNLOG line carrying a gate token AND an independent run-word (checked against the residue after tokens are stripped), or a parseable .loop/receipts/*.jsonl record. The genuinely gate-backed flagship (examples/coverage-repair) keeps its invoked credit — its double-quoted python3 "$REPO/scripts/holdout_gate.py" invocation is a real execution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…olders A fresh, unedited scaffold scored 86/100 "strong" — its success criteria and task titles are still the literal "REPLACE:" placeholders, so the inspector was crediting a shell as a defined success. A stranger's first `scaffold` then `inspect` wrongly read as a strong loop. The inspector now gates defines_success on the SPEC Success-criteria list being filled: a structurally-present-but-all-placeholder criteria list earns no credit and emits an actionable gap naming the "REPLACE:" convention. Placeholder TASKS titles are flagged the same way. Doctor is unchanged — a fresh scaffold is still a valid in-flight contract (doctor stays clean, a test-asserted invariant); only the inspector's *quality* verdict tightens. A fresh scaffold now lands at 74/ok with placeholder gaps; the flagship (examples/coverage-repair), which has real criteria, keeps 90/strong. Combined with the false-completion execution-evidence fix, the review's keyword-stuffed fake (stuffed prose + echo verify line + stuffed RUNLOG, no real gate) drops from 100/strong to 74/ok with a false-completion gap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…w strong Close four inspector-honesty holes the adversarial verifier flagged: 1. Genuine-invocation ALLOWLIST for false-completion "invoked" credit (_gate_invoked_in_verify): credit only when an interpreter (python/python3/python3.N/uv/bash/sh/exec) runs a gate script named in its args, or the gate script is invoked directly by path. grep/cat/ls/test/head/ wc/find/echo/printf that merely reference the .py now earn ZERO invoked credit — the old inert-emitter denylist let all of them evade. 2. Score cap: when false-completion defense grades "none", the total is capped at 79 with an explicit gap, so "strong" (>=80) and a fail-under-80 CI gate are unreachable by pure keyword stuffing. "wired"/"invoked" grades are uncapped. 3. independent_verification substance: a verify-* script earns credit only if it has >=1 non-inert executable line — a file merely named verify-fast whose body is echo/printf/no-ops is not verification. The shipped scaffold's for-loop existence check keeps its credit. 4. Adversarial regressions: grep/cat/ls/test/head/wc/find evasions, the grep-swapped M3 fake, the stuffed fake crossing 80, echo-only verify bodies, and the allowlist true positives. Flagship (examples/coverage-repair) holds 90/strong; fresh scaffold holds 74/ok and stays doctor-clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s, uv subcommands, ghost gates, prose run-words Blockers from the round-2 adversarial verify: - a gate named only in a trailing shell comment, as a redirection sink, or in a uv non-run subcommand (pip install/add) no longer earns invoked credit - invoked credit for a workspace-relative gate path now requires the gate file to exist on disk (full credit no longer has a weaker precondition than the half-credit wired grade); $VAR/absolute paths keep shape-only credit - RUNLOG credit requires the gate .py path plus a verdict word; ordinary English run-words (ran/result/passed-as-prose/clean) removed from the set - honest wrapper prefixes (env/time/nohup/nice/command) stay transparent
… verdict tokens, records need a gate on disk - redirection expressions are stripped BEFORE segment splitting, so >&/>|/&> sinks can no longer sever into a bare-path segment that reads as a direct gate invocation; a real invocation with a 2>&1 suffix keeps credit - verdict words match on word boundaries: cleanup/passphrase/surpassed no longer satisfy the clean/pass bar in RUNLOG prose or receipt JSON - RUNLOG/receipt record credit requires a gate script somewhere on disk — a record-shaped line about a tool that does not exist earns nothing
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6a7d41f6e8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| args = args[1:] | ||
| elif not _is_gate_interpreter(lead): | ||
| return False | ||
| return any(_gate_arg_credits(workspace, arg) for arg in args) |
There was a problem hiding this comment.
When a verify script uses an interpreter for a non-executing operation such as python3 -m py_compile scripts/holdout_gate.py or bash -n scripts/holdout_gate.py, this any(...) gives full invoked false-completion credit just because the gate path is anywhere in the arguments. I checked the CLI semantics locally: python3 -m py_compile --help describes the positional filenames as “Files to compile”, and bash -c 'help set' says -n means “Read commands but do not execute them”, so these cases only syntax/compile-check the gate and do not run it.
Useful? React with 👍 / 👎.
| tokens = tokens[1:] | ||
| if not tokens: | ||
| return False | ||
| lead, args = tokens[0], tokens[1:] |
There was a problem hiding this comment.
For verify scripts that run the gate with common inline environment setup, e.g. PYTHONPATH=. python3 scripts/holdout_gate.py, the first token is treated as the command name, so the detector rejects the line before it ever sees the interpreter. That incorrectly downgrades a genuine gate execution to wired/no-run evidence and can under-score otherwise valid loops that need env vars to import the gate or target code.
Useful? React with 👍 / 👎.
Comment on lines
+413
to
415
| lead = _leading_command(stripped) | ||
| if lead and lead not in _INERT_LINE_COMMANDS: | ||
| return True |
There was a problem hiding this comment.
A verify script that contains only standard shell setup plus output, such as set -euo pipefail followed by echo PASS, is currently counted as independent verification because set is not in _INERT_LINE_COMMANDS. help set describes this builtin as changing shell options/positional parameters, so in this context it does not verify the loop; this leaves the echo-only proof-surface bypass open as soon as the script includes normal shell boilerplate.
Useful? React with 👍 / 👎.
SollanSystems
added a commit
that referenced
this pull request
Jul 8, 2026
* feat(st2): the contract is a versioned, tool-agnostic standard Promote reference/repo-os-contract.md to the normative spec: stability note and versioning model (§0), artifact/schema table across all 7 published schemas with required keys verbatim from schemas/ (§11), lifecycle vocabulary + terminal-file-iff rule (§12), repair-record vs rollout-record two-shape clarification (§13), and the A1-E1 conformance checklist (§14). - doctor lifecycle line: validate_contract reports lifecycle: planned | running | terminated:<State> | unknown — additive reporting only, never an issue source; DG-3 regressions pin both directions in both validation modes. - scripts/test_template_roundtrip.py: every templates/* artifact, filled with schema-valid values, passes validate_contract with zero issues in both modes (in-flight + terminated scaffolds). - scripts/test_conformance.py: executes checklist A1-E1 in CI against examples/coverage-repair and a fresh template scaffold, incl. additive-key tolerance (D2), lifecycle honesty (E1), and a doc-parity guard binding every checklist ID to the normative doc. - README: pointer subsection for the versioned standard. ST2 spec: docs/superpowers/specs/2026-06-30-st2-portable-contract-spec.md (DG-1/DG-2/DG-3-core/M5/QW11 had already landed via #27-#30). Suite: 372 passed / 10 skipped (jsonschema), 361 / 21 (structural). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(release): cut 0.7.0 — portable standard CHANGELOG 0.7.0 collects the ST2 standard work, the adoption slices (A1/B1/C1/PR5, previously Unreleased), and the external-review patch set #27-#30 (previously unchangelogged). Version 0.6.1 -> 0.7.0 in pyproject.toml, plugin.json, README badge + Status; docs-version gate updated. The README Adopt-section @v0.7.0 action pin becomes accurate at this release's tag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the last two confirmed findings from the 2026-07-05 external review — the inspector-scoring exploits M2 and M3. With #27 (F1/F5/F6/F2/F7), #28 (F4), and #29 (F3/F8) already merged, this completes the 10-finding patch set.
false-completion defense (invoked)credit now requires a verify-* line that genuinely executes a gate script: an allowlisted interpreter (python/python3/python3.N/uv/bash/sh/exec) with the gate named in its arguments, or a direct by-path invocation.echo "holdout_gate", and the follow-up round'sgrep/cat/ls/test/head/wc/findreferences, earn nothing.RUNLOG/receipt credit requires a gate token AND an independent run-word on one line; receipts must parse as JSON.echo/printf/exit/true/false/:and comments earns no independent-verification credit — a file merely namedverify-fastis not verification.noneis capped at 79, below the 80 "strong" threshold and the Action's defaultfail-under-score, with an actionable gap. Keyword stuffing alone can no longer clear a CI inspect gate.REPLACE:success criteria earn nodefines_successcredit and all-placeholder TASKS titles are flagged; a fresh unedited scaffold now inspects 74/ok (was 86/strong) while staying doctor-clean (test-pinned invariant).A second adversarial verify round against the fix itself found three more laundering paths, closed in the final commit:
# comment, used as a redirection sink (python3 x.py > scripts/holdout_gate.py), or passed to a uv non-run subcommand (uv pip install …) earns nothing; onlyuv runexecutes.$VAR, absolute) keep shape-only credit, preserving the flagship'spython3 "$REPO/scripts/holdout_gate.py"..pypath plus a verdict word; ordinary English (ran,result, "the deadline passed", "keep it clean") no longer counts. Honest wrapper prefixes (env/time/nohup) stay transparent.A third round against those fixes found two operator-variant residuals, closed in the final commit:
>&/>|/&>contain segment-split characters, so a severed sink read as a bare-path invocation; redirection expressions are now stripped before segment splitting (a real invocation with a2>&1suffix keeps credit).cleanup/passphrase/surpassedsatisfied theclean/passbar; verdict vocabulary now matches whole words only, and RUNLOG/receipt record-credit requires a gate script to exist on disk at all.Verification
examples/coverage-repairkeeps 90/strong with(invoked); chained (echo x && python3 …), quoted (python3 "$REPO/…"), wrapper-prefixed, anduv runreal invocations keep credit.Known, documented limitation
An author who fabricates a gate file (empty
scripts/holdout_gate.py+ a SPEC reference) still earnswiredhalf-credit and can reach strong. That is deliberate: per the #29 docs,loop inspectis an advisory heuristic a determined author can game —loop doctoris the hard gate. The cap targets loops with no defense at all, not active fraud.🤖 Generated with Claude Code