Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions examples/phase2-final/dogfood-round2/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Dogfood Round 2 repair evidence

This directory records the local Round 2 repair of the failed Phase 2
dogfood trial. It contains exported, non-canonical evidence only. Canonical
Benchwork state was changed through the typed MCP boundary and no
`.benchwork/` state file is included here.

## Root cause

Round 1 attempted to complete `EX-001` with a run plan and metric that did not
match frozen Protocol `PT-001`. The Protocol required `RUN-000` and metric
`score`, while the trial recorded `RUN-001` through `RUN-005` using
`replay_success`. Athanor rejected pilot completion with
`VALIDATION_REJECTED`, and the failed trial was preserved.

## Repair

The repair preserved all Round 1 records. In particular, frozen `PT-001`,
active `WK-001`, `EX-001`, and their Chronicle histories remain unchanged.
The shared `RP-001`, `CL-002`, and `HY-002` aggregate objects advanced through
the registered evaluation while the repair added a new path:

```text
PT-002 -> WK-002 -> AR-002 -> EX-002
-> RUN-101..RUN-105 -> RB-001 -> AS-001 -> DE-001
```

- `PT-002` freezes the included Pilot set as `RUN-101` through `RUN-104`.
- `RUN-105` remains a failed, excluded negative result with an explicit reason.
- `RB-001` analyzes only the four completed, included runs.
- `AS-001` records the synthetic-host limitations.
- `DE-001` seals the user-confirmed `REPAIR` outcome.
- `WK-002` finishes as `COMPLETED`; the original `WK-001` and `EX-001`
remain in their failed-trial states.

The paired `replay_success` comparison produced a treatment mean of `1`, a
control mean of `0`, and a mean difference of `1` with a 95% Student-t
interval of `[1, 1]`. The standardized effect is unavailable because paired
differences have zero variance.

## Integrity checks

The final Deep Doctor report passes with 53 verified Chronicle events and 37
replayable objects. A fresh process restart exported the same status bytes
before and after restart:

```text
2d586f3dbb364808da96c1b212bb3371735a022cbdb0c9a226434f6c08aa64a2
```

Run the local evidence checks from the repository root:

```bash
sha256sum --check examples/phase2-final/dogfood-round2/SHA256SUMS
cmp \
examples/phase2-final/dogfood-round2/round2-before-restart.json \
examples/phase2-final/dogfood-round2/round2-after-restart.json
jq -e \
'.ok and .chronicle_verified and .all_objects_replayable' \
examples/phase2-final/dogfood-round2/restart-doctor.json
```

The Round 1 Protocol and Experiment traces are included so preservation can be
audited independently of the final aggregate snapshots. Full interactive
command logs are intentionally omitted because they are noisy and are not
required to verify the registered result.
9 changes: 9 additions & 0 deletions examples/phase2-final/dogfood-round2/SHA256SUMS
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
ed3e2d99a7ae7e32f19d1bbacebd73e88c6c2fa8f2c50cd3e2091158446d92ec examples/phase2-final/dogfood-round2/analysis-spec-round2.json
3e355f4d64d22dd25adfa215463b0a6d26b499249229496a26a1d34759cfefe5 examples/phase2-final/dogfood-round2/implementation-artifact-round2.json
ebc306d07eee42cc433829814bef1b6d5c69f2def0a674e170ebde62d926f331 examples/phase2-final/dogfood-round2/restart-doctor.json
be29cf6487425fe5a5712c3fb347f061f8f06d32a358359a452d1c1ab94ac832 examples/phase2-final/dogfood-round2/restart-verify.txt
3579161fda803e46a0b871d0162826fca025e9133d5bc772e913826023613867 examples/phase2-final/dogfood-round2/round1-ex001-trace.json
8b2e437d43a182e346e6bb745f32eafc6de0472eb71d75687058f67a3e41f79c examples/phase2-final/dogfood-round2/round1-final-status.json
b929e98a6730a15bf7a6e13972a2b7224ff198a399c9268f69458d348f9141fe examples/phase2-final/dogfood-round2/round1-pt001-trace.json
2d586f3dbb364808da96c1b212bb3371735a022cbdb0c9a226434f6c08aa64a2 examples/phase2-final/dogfood-round2/round2-after-restart.json
2d586f3dbb364808da96c1b212bb3371735a022cbdb0c9a226434f6c08aa64a2 examples/phase2-final/dogfood-round2/round2-before-restart.json
35 changes: 35 additions & 0 deletions examples/phase2-final/dogfood-round2/analysis-spec-round2.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"schema_version": "analysis-spec/1.0",
"comparisons": [
{
"comparison_id": "CMP-002",
"experiment_id": "EX-002",
"arms": [
"baseline",
"treatment"
],
"metric": "replay_success",
"estimand": "mean_difference",
"pairing": "paired",
"uncertainty_method": "student_t",
"confidence_level": 0.95
}
],
"multiple_comparison_policy": "none",
"practical_significance_thresholds": {
"replay_success": 0.0
},
"pilot_run_ids": [
"RUN-101",
"RUN-102",
"RUN-103",
"RUN-104"
],
"expected_run_ids": [
"RUN-101",
"RUN-102",
"RUN-103",
"RUN-104",
"RUN-105"
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"schema_version": "dogfood-implementation-artifact/1.0",
"name": "benchwork-dogfood-replay-repair",
"version": "0.2",
"purpose": "Validate canonical replay through a Protocol-aligned registered pilot and preserve the excluded failed Run.",
"protocol_id": "PT-002",
"working_id": "WK-002",
"analysis_spec": "analysis-spec-round2.json",
"experiment_id": "EX-002",
"required_pilot_run_ids": [
"RUN-101",
"RUN-102",
"RUN-103",
"RUN-104"
],
"preserved_failed_run_ids": [
"RUN-105"
]
}
57 changes: 57 additions & 0 deletions examples/phase2-final/dogfood-round2/restart-doctor.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
{
"schema_version": "doctor-report/1.1",
"mode": "deep",
"ok": true,
"chronicle_verified": true,
"all_objects_replayable": true,
"checks": {
"chronicle": {
"status": "PASS",
"event_count": 53
},
"projection": {
"status": "PASS",
"object_count": 37
},
"snapshots": {
"status": "PASS",
"verified_count": 6
},
"capsules": {
"status": "PASS",
"verified_count": 6
},
"capability_registry": {
"status": "PASS",
"verified_count": 21
},
"rite_registry": {
"status": "PASS",
"verified_count": 3
},
"grimoire_registry": {
"status": "PASS",
"verified_count": 0
},
"agent_outputs": {
"status": "PASS",
"verified_count": 3
},
"artifacts": {
"status": "PASS",
"verified_count": 2
},
"result_exports": {
"status": "PASS",
"verified_count": 1
},
"project_context": {
"status": "PASS",
"active_program_id": "RP-001"
},
"migration_state": {
"status": "PASS",
"residual_count": 0
}
}
}
1 change: 1 addition & 0 deletions examples/phase2-final/dogfood-round2/restart-verify.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Chronicle healthy: 53 verified event(s), receipt chain intact
Loading