Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ versioned schemas plus explicit migration notes for compatibility.

## 0.3.0rc2 — The Invitation

- Fixed `benchwork_open_task` minting every Task Capsule for the `codex` Host.
It now accepts the optional `host_session` argument the completion tools
already took, so a non-Codex Host can record truthful provenance. The default
remains `codex` and an unregistered Host is refused.
- Recorded the Claude Code CLI Tier 1 Host PASS from its 2026-07-31 trial.
- Added the auditable POSIX installer and exact release/channel manifests.
- Added uv-first isolated installation, pipx fallback, plugin staging,
installer state, Host diagnostics, repair, rollback, and uninstall.
Expand Down
36 changes: 31 additions & 5 deletions docs/en/HOST_SUPPORT_MATRIX.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ tier.

Tier 1 is evaluated per declared release Host. Codex CLI is the primary Phase
2 Host. Host-neutral compatibility with another CLI is not the same as an
interactive PASS for that Host.
interactive PASS for that Host; Claude Code CLI earned its own PASS from its
own trial, not from the shared contract.

## Current acceptance

Expand All @@ -29,7 +30,7 @@ interactive PASS for that Host.
| Core Runtime | 0 | PASS | Required gate satisfied |
| MCP STDIO | MCP | PASS | Required gate satisfied |
| Codex CLI | 1 | PASS | Primary CLI gate satisfied |
| Claude Code CLI | 1 | PENDING_HOST_VALIDATION | Host-neutral contract only; not a claimed interactive PASS |
| Claude Code CLI | 1 | PASS | Second CLI gate satisfied by the 2026-07-31 trial |
| Codex IDE extension | 2 | BLOCKED_BY_ENVIRONMENT | Accepted exception `HOST-IDE-001` |
| Claude Desktop | 2 | OPTIONAL_NOT_RUN | No Phase 2 release impact |
| External Review | disclosure | WAITING_FOR_DISCLOSURE_AUTHORIZATION | Correctly denied; not a failure |
Expand Down Expand Up @@ -57,12 +58,37 @@ Reproduction and closure:
6. Record Host version, plugin version, tool count, Receipt identifier, and
any exception without copying secrets or conversation reasoning.

## 2026-07-31 Claude Code CLI trial

Fresh ephemeral Claude Code CLI `2.1.220` conversations loaded the Benchwork
MCP server with `--strict-mcp-config` against a temporary project outside the
source repository. The server reported `connected` and the Host discovered all
38 registry tools. `benchwork_status` and `benchwork_next_actions` each
returned `ok: true` with an `mcp-tool-result/1.0` envelope, both from an
explicit request and from the broad request "What should we do next for this
study?" with 26 non-MCP tools still available.

The bounded Task loop initially failed. `benchwork_open_task` minted every Task
Capsule with a hardcoded `codex` Host, so a truthful `claude-code` provenance
was refused with `VALIDATION_REJECTED`. Athanor was correct; the MCP boundary
gave a non-Codex Host no way to describe itself. After `benchwork_open_task`
was given the same optional `host_session` argument the completion tools
already accepted, the loop completed: Capsule `TK-B280B5160682` recorded host
`claude-code`, Ward returned `PASS`, and Athanor issued Receipt
`RC-47D74468807A` for the accepted `code-inspection-result/1.0` output. The
source repository never acquired a `.benchwork/` directory.

Evidence is published under
[`examples/phase2-final/host-claude-code`](../../examples/phase2-final/host-claude-code/README.md).

## Evidence policy

PASS requires a dated, reproducible Host trial. Contract inspection or shared
configuration alone is insufficient. `BLOCKED_BY_ENVIRONMENT` requires an
identifier, owner, reason, impact, reproduction steps, and the boundary before
which it must be closed.
configuration alone is insufficient. The Claude Code CLI trial demonstrates
why: the adapter, the Capsule schema, and Ward all already admitted
`claude-code`, and only a live Host reached the one line that did not.
`BLOCKED_BY_ENVIRONMENT` requires an identifier, owner, reason, impact,
reproduction steps, and the boundary before which it must be closed.

See the [Phase 2 acceptance matrix](PHASE2_ACCEPTANCE.md) and
[Acceptance Exception Policy](plugins/acceptance-exception-policy.md).
29 changes: 28 additions & 1 deletion docs/en/PHASE2_ACCEPTANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ states.
| Tier 0 — Kernel | PASS | Required | Chronicle, Schema, Ward, Capability, replay, and deterministic suites |
| MCP | PASS | Required | in-memory and spawned-STDIO discovery, envelopes, pagination, Task and canonical tool tests |
| Tier 1 — Codex CLI Host | PASS | Required before Phase release | fresh Codex CLI explicit/implicit Skill trials and MCP calls |
| Tier 1 — Claude Code CLI | PENDING_HOST_VALIDATION | Not a claimed Phase 2 Host PASS | host-neutral contract is implemented; independent interactive evidence is pending |
| Tier 1 — Claude Code CLI | PASS | Second CLI Host gate satisfied | fresh Claude Code CLI MCP discovery, read-only calls, and a bounded Task loop with an Athanor Receipt |
| Tier 2 — IDE Host | BLOCKED_BY_ENVIRONMENT (accepted) | Does not block Core/MCP/CLI | `HOST-IDE-001`; extension installed, no graphical extension Host available |
| External Review | WAITING_FOR_DISCLOSURE_AUTHORIZATION | Does not block local review or Core/MCP/CLI | Disclosure Gate denied an unapproved diff review |

Expand Down Expand Up @@ -79,6 +79,33 @@ an accepted `bench.study.audit` proposal, replacement Protocol preview/commit,
registered implementation and Runs, Alembic analysis, Assessment, and a
human-sealed `REPAIR` Decision through Chronicle replay.

## 2026-07-31 Claude Code CLI Tier 1 trial

- Fresh ephemeral Claude Code CLI `2.1.220` conversations loaded the Benchwork
MCP server with `--strict-mcp-config` against a temporary project outside the
source repository. The server reported `connected` and all 38 registry tools
were discovered.
- `benchwork_status` and `benchwork_next_actions` both returned `ok=true` with
`mcp-tool-result/1.0`, from an explicit request and from the broad request
"What should we do next for this study?" while 26 non-MCP tools remained
available.
- The first bounded Task loop was blocked. `benchwork_open_task` minted the
Capsule with a hardcoded `codex` Host, so truthful `claude-code` provenance
was refused with `VALIDATION_REJECTED`. The Host reported the rejection
rather than resubmitting a false provenance.
- `benchwork_open_task` now accepts the optional `host_session` argument that
the completion tools already took, defaulting to `codex` and refusing an
unregistered Host. The repaired loop completed `bench.code.inspect` as
`TK-B280B5160682` with Capsule host `claude-code`, Ward `PASS`, and Receipt
`RC-47D74468807A`.
- The Host used only `Read` alongside the Benchwork tools, matching the Capsule
Circle. Deep Doctor reported 2 verified Chronicle events with an intact
Receipt chain, and the source repository had no `.benchwork/` directory.
- Evidence is published under
[`examples/phase2-final/host-claude-code`](../../examples/phase2-final/host-claude-code/README.md).
- IDE execution and External Review were not exercised; `HOST-IDE-001` and
`WAITING_FOR_DISCLOSURE_AUTHORIZATION` are unchanged.

## 2026-07-30 M17 rc1 trial

- The Python wheel and source distribution built as `0.3.0rc1`; Twine accepted
Expand Down
12 changes: 8 additions & 4 deletions docs/en/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,10 +127,14 @@ audit-to-`REPAIR` acceptance; migration and release documentation; and
release-gated CI coverage.

Phase 2 is sealed with Kernel, MCP, Plugin, and Codex CLI gates passing.
`HOST-IDE-001` remains an accepted `BLOCKED_BY_ENVIRONMENT` Tier 2 exception,
External Review remains `WAITING_FOR_DISCLOSURE_AUTHORIZATION`, and Claude Code
CLI remains `PENDING_HOST_VALIDATION`. These states are not reclassified as
failures or fabricated PASS results.
`HOST-IDE-001` remains an accepted `BLOCKED_BY_ENVIRONMENT` Tier 2 exception
and External Review remains `WAITING_FOR_DISCLOSURE_AUTHORIZATION`. These
states are not reclassified as failures or fabricated PASS results.

Claude Code CLI was `PENDING_HOST_VALIDATION` at the rc1 seal and recorded its
own Tier 1 PASS on 2026-07-31, after its trial found and repaired a Task
Capsule Host defect. See the
[Host Support Matrix](HOST_SUPPORT_MATRIX.md).

## M18: The Invitation

Expand Down
6 changes: 4 additions & 2 deletions docs/en/plugins/acceptance-exception-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,10 @@ An unavailable Tier 2 environment is recorded as `BLOCKED_BY_ENVIRONMENT`, not
the release boundary before which validation becomes required.

Tier 1 is accepted per Host. Codex CLI is the Phase 2 primary Host and has a
recorded PASS. Claude Code CLI remains `PENDING_HOST_VALIDATION`; the
host-neutral tool contract does not manufacture an interactive Host result.
recorded PASS. Claude Code CLI recorded its own PASS on 2026-07-31 from its own
interactive trial; the host-neutral tool contract does not manufacture an
interactive Host result, and that trial found and repaired a Task Capsule Host
defect the shared contract had hidden.

The current IDE exception is:

Expand Down
22 changes: 20 additions & 2 deletions docs/en/releases/0.3.0rc2.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@ canonical: true

# Benchwork 0.3.0rc2 — The Invitation

`0.3.0rc2` adds the auditable macOS/Linux/WSL installer without changing the
M17 scientific control-plane contracts.
`0.3.0rc2` adds the auditable macOS/Linux/WSL installer. The M17 scientific
control-plane contracts are unchanged apart from one additive optional field,
recorded under Fixed below.

## Added

Expand All @@ -19,6 +20,23 @@ M17 scientific control-plane contracts.
- installer-owned repair, rollback, PATH blocks, and uninstall
- release assets, SBOM, provenance, GitHub Pages publication, and installer CI

## Fixed

`benchwork_open_task` minted every Task Capsule for the `codex` Host, so a
non-Codex Tier 1 Host could not record truthful provenance: Athanor correctly
refused the Agent Result with `VALIDATION_REJECTED`. The tool now accepts the
same optional `host_session` argument `benchwork_complete_task` and
`benchwork_fail_task` already took, and mints the Capsule for that Host.

This is an additive optional field under the
[MCP API Policy](../MCP_API_POLICY.md). The 38 frozen tool names are unchanged,
the default remains `codex` so existing clients behave exactly as before, and
an unregistered Host is now refused at the boundary instead of being minted and
later rejected by Ward.

The Claude Code CLI Tier 1 trial that found this is published under
[`examples/phase2-final/host-claude-code`](../../../examples/phase2-final/host-claude-code/README.md).

## Safety

The installer does not require root, mutate system Python, create research
Expand Down
99 changes: 99 additions & 0 deletions examples/phase2-final/host-claude-code/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Claude Code CLI Tier 1 Host trial

This directory records the 2026-07-31 Tier 1 Host validation for Claude Code
CLI, which closed the `PENDING_HOST_VALIDATION` state left open by the Phase 2
acceptance. It contains exported, non-canonical evidence only. No `.benchwork/`
state file and no conversation reasoning is included here.

## Method

Each trial was a fresh, ephemeral Claude Code CLI `2.1.220` conversation
started with `--strict-mcp-config`, so the Benchwork MCP server was the only
one loaded. Every trial ran against a temporary project outside this
repository; the source repository never acquired a `.benchwork/` directory.

Transcripts were captured as `stream-json` and distilled into
[`trial-log.json`](trial-log.json). The Host session identifier was pinned with
`--session-id`, so the `invocation_id` recorded in Chronicle can be checked
against the transcript that produced it.

## What the trial found

Read-only discovery passed immediately. The MCP server reported `connected`
and the Host discovered all 38 registry tools. `benchwork_status` and
`benchwork_next_actions` each returned `ok: true` with an
`mcp-tool-result/1.0` envelope, both when the tools were named explicitly and
when the Host was given only the broad request "What should we do next for this
study?" with 26 non-MCP tools still available.

The bounded Task loop did not pass. `benchwork_open_task` minted every Task
Capsule with a hardcoded `codex` Host and exposed no parameter for the acting
Host, so a truthful `claude-code` provenance was refused:

```text
VALIDATION_REJECTED: Agent Result provenance Host does not match its Task Capsule
```

Athanor was right to refuse. The defect was that the MCP boundary gave a
non-Codex Host no way to describe itself, in a module whose own docstring
claims to be host-neutral. The Host reported the rejection rather than
resubmitting a false `codex` provenance to force a green run.

This is what the evidence policy means when it says contract inspection is
insufficient. `hosts.py` already exported a `ClaudeCodeHostAdapter`, the Task
Capsule schema already enumerated `claude-code`, and Ward already admitted it.
Only a live Host trial reached the one line that did not.

## The repair

`benchwork_open_task` now accepts the same optional `host_session` argument
that `benchwork_complete_task` and `benchwork_fail_task` already took, and
mints the Capsule for that Host. The default remains `codex`, so existing
Codex flows are unchanged, and an unregistered Host is refused rather than
minted and later rejected by Ward.

## Result

The repaired loop completed end to end. Chronicle recorded the Capsule and the
accepted Agent Result under the acting Host:

| Item | Value |
| --- | --- |
| Program | `RP-001`, Receipt `RC-3913CD0AC6A6` |
| Task | `TK-B280B5160682`, capability `bench.code.inspect` |
| Capsule Host | `claude-code` |
| Circle | `read` only, no network, 900 s |
| Ward | `PASS` |
| Completion | `ok: true`, Receipt `RC-47D74468807A` |
| Output | `code-inspection-result/1.0` |

The Host used only `Read` alongside the Benchwork tools, matching the Capsule
Circle. Deep Doctor reported 2 verified Chronicle events with an intact Receipt
chain. The canonical provenance excerpt is in
[`agent-result-provenance.json`](agent-result-provenance.json).

## Verify

```bash
sha256sum --check examples/phase2-final/host-claude-code/SHA256SUMS

jq -e '.verdict.tier_1_claude_code_cli == "PASS"' \
examples/phase2-final/host-claude-code/trial-log.json

jq -e '.agent_result.provenance.host == "claude-code"' \
examples/phase2-final/host-claude-code/agent-result-provenance.json

python3 -m unittest \
tests.mcp.test_runtime.MCPRuntimeTest.test_declared_host_owns_its_capsule_and_completes_the_task \
tests.mcp.test_runtime.MCPRuntimeTest.test_capsule_defaults_to_codex_and_rejects_a_foreign_host_result \
tests.mcp.test_runtime.MCPRuntimeTest.test_unknown_task_host_is_refused
```

## Scope

This trial closes Tier 1 for Claude Code CLI only. `HOST-IDE-001` remains
`BLOCKED_BY_ENVIRONMENT` and External Review remains
`WAITING_FOR_DISCLOSURE_AUTHORIZATION`; neither was exercised here. The
inspection output also recorded a residual risk worth carrying forward: the
Circle preservation requirement in the Host adapter README is documentation
prose, not an enforced mechanism.
2 changes: 2 additions & 0 deletions examples/phase2-final/host-claude-code/SHA256SUMS
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
f032311827150fa5a728df72ea1842b0aa32f6c02e978d4b0b7f038a4e144f45 examples/phase2-final/host-claude-code/agent-result-provenance.json
56e8b4a0b59e5d4eb22606fd7a09aff6237d24bcb7755f4c461de836d0bf250a examples/phase2-final/host-claude-code/trial-log.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{
"agent_result": {
"outputs": [
{
"blob_sigil": "sha256:f718355fa44902306d2b0680e940769b99c19a91b3f42af56ae4a74faea10860",
"schema": "code-inspection-result/1.0",
"uri": ".benchwork/mcp/task-results/TK-B280B5160682-f718355fa449.json"
}
],
"provenance": {
"host": "claude-code",
"invocation_id": "44444444-4444-4444-8444-444444444444",
"model": "claude-opus-5",
"provider": "anthropic",
"runtime": "claude-code-cli/2.1.220"
},
"schema_version": "agent-result/1.1",
"status": "COMPLETED"
},
"capsule_host": "claude-code",
"event_id": "CE-8E61DDA3265E",
"exported_from": "Chronicle event payload of the round B Task completion",
"object_id": "TK-B280B5160682",
"occurred_at": "2026-07-31T08:46:31.240296+00:00"
}
Loading