Promote the experiment probe into the package behind agentseam probe

[jothimani-rajendran]

What this changes

The eight-trial probe moves out of tools/ into src/agentseam/probe/ and gets a CLI verb, so a pip install agentseam can measure what its own agent's hooks actually enforce — previously the harness existed but shipped to nobody. Adds a CI gate that replays committed recordings and fails on disagreement, and a documented basis vocabulary (witnessed / tested / recorded).

Wave 1 of armed (idea 01 of the portfolio direction); brief and contract in org-plan at plan/armed/, pinned at 83af785.

Claim check

• No capability claim is widened without a mechanism behind it
• Any new/changed MATRIX row carries a verified record — no matrix row is added or changed by this PR
• Payload shapes come from a primary source — unchanged; this PR moves and packages existing measurement code, and adds no new payload shape

No row is labelled witnessed here, and no measurement is synthesized. The coverage the evidence actually supports is unchanged: seven trials, at pre_tool, for one agent. The three gaps (escalate at pre_tool, all trials at prompt_submit, all trials at stop) ship as docs/witness-skeleton.json — copy-paste commands plus a blank report skeleton — with tests/test_coverage_skeleton.py asserting the skeleton is rejected by evidence_report.validate() while still blank, and validates once genuinely filled. Only the owner's machine can fill them; this branch was built with no vendor credentials.

Checks

• pytest -q passes — 1615 passed, 4 skipped (3.11 and 3.13)
• ruff check . and ruff format --check . pass — All checks passed! / 147 files already formatted
• Runtime path is still stdlib-only — dependencies = [] untouched, no new imports outside the standard library
• Commits are signed off

Notes for the reviewer

One defect found and fixed, and it is the one worth looking at. experiment_report.render() counted disagreements, printed "the matrix and this agent do not match", and then return 0. Harmless while it was a dev script; not harmless the moment the same code becomes a shipped agentseam probe run. A verb that reports a mismatch and tells its caller it succeeded is the exact shape this probe exists to catch in other people's tooling. Reproduced against the committed recording with claude_code.events.pre_tool.block flipped — exit 0 before, exit 1 after, agreement still 0 — with both directions pinned by tests so the gate cannot start crying wolf.

Two other things were deliberately made loud rather than convenient. tools/probe_ci.py reports the checked count beside the disagreement count and prints no recordings are committed -- this is a measurement of nothing, not a pass when there is nothing to check, so an empty run cannot read as green. agentseam probe run --driver recorded with no --trial filter raises NoRecording: claude_code@2.1.263/pre_tool never recorded the 'escalate' trial rather than quietly reporting seven of eight — correct, but it means the default recorded invocation fails until escalate is witnessed. Worth a decision: fail loudly as now, or report per-trial coverage with the unrecorded ones marked absent.

Naming is open. probe was chosen over "armed" because the CLI already speaks in verbs (agents, matrix, doctor, install) — the initiative's name reads oddly as one. Owner's call.

Promotion into src/ put the harness under the strict lint baseline for the first time. That was handled by fixing rather than ignoring: modules split by activity to stay inside the line budget, explicit check=False on every subprocess.run, a complexity split in reference_agent._interpret reusing existing tests unchanged. per-file-ignores were added only for pre-existing load-bearing shapes, each with a stated reason. One real correctness bug surfaced on the way: the generated probe script assumed a ../src checkout layout to find agentseam, which is false in an installed wheel — now derived from agentseam.__file__ and verified by running a trial end to end from a built wheel in a clean venv with no source checkout on sys.path.

🤖 Generated with Claude Code

---

Generated by Claude Code

[jothimani-rajendran]

semgrep is red, and it is this PR's — but the fix is a decision, not a keystroke

Failing check: semgrep, 4 findings of agentseam-no-shell-true, all Blocking:

• src/agentseam/probe/experiment_driver.py:29
• src/agentseam/probe/reference_agent.py:148, :203, :237

Why it went red, precisely. .github/workflows/security.yml:71 scans src/ tests/ — not tools/. On main, git grep -c shell=True -- src/ returns nothing and tools/ returns exactly these four (experiment_driver.py 1, reference_agent.py 3). T1's promotion did not introduce a single shell=True; it moved four pre-existing ones from an unscanned directory into a scanned one. The rule is working correctly. This is the same story as the lint baseline in the PR body: promotion put this code under the project's real standards for the first time.

Why the rule's own prescribed fix is the wrong one here. The message says "pass an argv list instead." For this code that would be actively harmful:

• reference_agent.py exists to be Claude Code's protocol made executable. Claude Code runs hooks by handing a command string to a shell. Emulating that with an argv list would make the emulation false, and this driver is the thing every tested-basis row is measured against — a false emulation publishes wrong conclusions about vendors with total confidence.
• :203 and :237 run pre["updated"] or command: the command as rewritten by the hook. That is the transform trial, whose entire purpose is measuring whether a rewritten input is what actually runs. Rewriting it as argv would measure something else.
• experiment_driver.py:29 runs the operator's own --driver template with {prompt} substituted — a command line the operator typed on their own machine.

So shell=True is intrinsic to what this package does, while the rule remains correct for the surface it was written for: .semgrep/agentseam.yml's sibling rules are scoped paths: include: ["src/agentseam/*"] with the rationale "code runs in the path of every tool call an agent makes". The probe is not that path — it is a deliberate command-runner that ships beside it.

Not fixing this by silencing it. no-weakening-the-check is a blocking gate, and this file's own header says "never delete one without a decision record." Narrowing a security rule is the owner's call, so this PR stays red until that call is made rather than going green by suppression.

Recommended patch — one audited chokepoint, rule untouched and still global:

Add src/agentseam/probe/_shell.py with a single run_shell() carrying one nosemgrep: agentseam-no-shell-true and a written trust-boundary rationale; route all four call sites through it. Four scattered suppressions become one reviewable place, the justification lives beside the condition as CLAUDE.md requires, and the invariant keeps biting everywhere else in src/.

The alternative, if you would rather the rule state its own scope: add paths: include: ["src/agentseam/*"], exclude: ["src/agentseam/probe/*"] to agentseam-no-shell-true, matching the convention its two siblings already follow — plus the decision record the file's header demands.

Everything else on this branch is green: 1615 passed, 4 skipped on 3.11 and 3.13, ruff check . and ruff format --check . clean, DCO satisfied on all four commits.

---

Generated by Claude Code

[jothimani-rajendran]

Fixed at 0c510e1 — one audited chokepoint, rule untouched

Owner picked the chokepoint option. src/agentseam/probe/_shell.py now holds the single run_shell() carrying one nosemgrep and the trust-boundary rationale; all four call sites route through it. .semgrep/agentseam.yml is unmodified — no rule narrowed, no path excluded, no decision record needed.

Verified narrow, not blanket. A canary shell=True dropped into src/agentseam/probe/_canary.py — the same package as the exception — still trips agentseam-no-shell-true and exits 1; removing it returns to exit 0. The rule keeps biting everywhere in src/, including beside the one function that is allowed to do this.

Green on CI's own invocation:

semgrep --config .semgrep/agentseam.yml --error --quiet src/ tests/   → exit 0
pytest -q                     → 1615 passed, 4 skipped
ruff check .                  → All checks passed!
ruff format --check .         → 148 files already formatted

One thing worth noting from the fix: routing through the wrapper initially tripped PLW1510 (subprocess.run without explicit check), because kwargs.setdefault("check", False) is invisible to ruff. Fixed by making check and capture_output explicit keyword parameters rather than silencing the rule — the wrapper is now more readable at the call site too.

Probe re-checked end to end after the change: agreement exits 0, a forced disagreement prints DISAGREES and exits 1.

---

Generated by Claude Code