Skip to content

test(_util): match error ANNOUNCEMENTS, not the word "error" (unblocks #70) #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Brian Krabach (bkrabach) merged 2 commits into from
Sep 3, 2026
Merged

test(_util): match error ANNOUNCEMENTS, not the word "error" (unblocks #70) #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
34913a0
test(_util): match error ANNOUNCEMENTS, not the word "error"
Sep 3, 2026
2760376
docs(lane wp6): DONE-NOTE + fail-before, tier, and #70 evidence under…
Sep 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
338 changes: 338 additions & 0 deletions docs/lanes/wp6-error-regex-scope/DONE-NOTE.md
View file
Open in desktop

Large diffs are not rendered by default.

7 changes: 7 additions & 0 deletions docs/lanes/wp6-error-regex-scope/evidence/check.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
=== make check (lint + types) ===
.venv/bin/ruff check .
All checks passed!
.venv/bin/ruff format --check .
149 files already formatted
.venv/bin/pyright src tests
0 errors, 0 warnings, 0 informations
350 changes: 350 additions & 0 deletions docs/lanes/wp6-error-regex-scope/evidence/fail-before.txt
View file
Open in desktop

Large diffs are not rendered by default.

102 changes: 102 additions & 0 deletions docs/lanes/wp6-error-regex-scope/evidence/jyg-cannot-be-bypassed-from-outside.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
=== ATTEMPT (FAILED) to reach the tier-3 doctor test's silent-failure check locally ===

Hypothesis: run_cli does env.setdefault("AMPLIFIER_WORK_TRACKER_ROOT", ...),
so exporting the REAL root (/home/bkrabach/.amplifier-work-tracker) would let
the subprocess see this machine's live reap/notify heartbeats, sweeps.alive
would PASS, doctor would exit 0, and the test would finally REACH
assert_no_silent_failure -- the line wp6 is about.

RESULT: hypothesis is WRONG, and the reason matters.
tests/conftest.py:308 does mp.setenv("AMPLIFIER_WORK_TRACKER_ROOT", str(root))
at session scope, so os.environ ALREADY carries the isolated tmp root by the
time setdefault runs. An exported value cannot win. sweeps.alive still FAILs
(see the captured run below), doctor still exits 1, and the test still dies at
the earlier `assert result.returncode == 0`.

CONSEQUENCE, stated plainly: on any machine whose sweep heartbeats live
outside the isolated test root, this tier-3 test CANNOT reach the
silent-failure check at all, and no environment variable fixes that from
outside. That is model_performance-jyg, and it is exactly why the wp6 defect
was invisible locally and only surfaced in CI.

It is also why this lane's regression test is TIER 1: a tier-1 test of the
predicate itself needs no bd, no dolt, and no healthy sweeps, so it cannot be
masked this way. The end-to-end evidence for #70 was instead obtained by
running doctor --quick directly, outside pytest, against the real root
(see pr70-would-now-pass.txt).

--------------------------- the captured run ---------------------------
=== tier-3 doctor test, THIS BRANCH, in an environment where doctor exits 0 ===

The run_cli fixture uses env.setdefault("AMPLIFIER_WORK_TRACKER_ROOT", ...),
so exporting the REAL root makes the subprocess see this machine's live reap/
notify heartbeats -- i.e. sweeps.alive PASSES and doctor exits 0. That is the
healthy-CI condition, and the only condition under which the wp6 defect is
reachable at all (see jyg-preexisting.txt).

command: AMPLIFIER_WORK_TRACKER_ROOT=/home/bkrabach/.amplifier-work-tracker \
.venv/bin/python -m pytest -m cli tests/cli/test_cli_surface.py::test_doctor_quick_succeeds_against_the_real_installed_bd -q

F [100%]
=================================== FAILURES ===================================
___________ test_doctor_quick_succeeds_against_the_real_installed_bd ___________

run_cli = <function run_cli.<locals>._run at 0xfdfac5791080>

def test_doctor_quick_succeeds_against_the_real_installed_bd(run_cli):
result = run_cli(["doctor", "--quick"])
> assert result.returncode == 0, result.stdout + result.stderr
E AssertionError: [PASS] version bd 1.1.2
E [PASS] capabilities all required bd commands present
E [PASS] resolve.fenced stale holder refused, as required
E [PASS] resolve.divergent_text_refused resolving a closed item with different text refuses and writes nothing
E [PASS] resolve.identical_text_idempotent re-sending identical resolution text is an idempotent success
E [PASS] reopen.reopens a resolved item reopens unassigned and is directly claimable again
E [PASS] reopen.clears_closed_at reopen clears closed_at (the documented, surfaced accounting cost)
E [PASS] reopen.close_reason_disposition reopen clears close_reason (measured), and the wrapper's archive comment preserves the previous resolution regardless
E [PASS] reopen.emits_event bd records a `reopened` events row, attributed
E [PASS] release.reopens_unresolved release reopens a held item with no resolution, and it is re-claimable
E [PASS] claim.subcommand --claim present, rejects --assignee as expected
E [PASS] claim.atomic skipped (--quick); run full doctor before trusting parallel agents
E [PASS] claim.directed_atomic skipped (--quick); run full doctor before trusting parallel agents
E [PASS] link.nonblocking discovered-from is non-blocking
E [PASS] list.includes_closed all-flag required and working
E [PASS] list.status_filter_includes_closed an explicit --status filter shows closed items without --all
E [PASS] show.dependents reverse link visible (1 links)
E [PASS] read.no_mutation repeated reads (including not-found/wrong-project misses) leave status, holder, and metadata unchanged
E [PASS] resolution.readable resolution text round-trips
E [PASS] timestamps.readable created_at/updated_at/closed_at all round-trip as real datetimes
E [PASS] metadata.roundtrip arbitrary JSON metadata round-trips
E [PASS] project.name_rules dotted names appear usable now; validator may be relaxed
E [PASS] custody.fresh_survives a fresh renewal survives regardless of total hold duration
E [PASS] custody.stale_reclaimed stale custody is reclaimed: custody stale -- last seen 3600s ago (ttl 900s)
E [PASS] custody.idle_not_exempt awaiting_human with stale custody is still reclaimed: custody stale -- last seen 3600s ago (ttl 900s)
E [PASS] custody.fenced old holder's renew and resolve are both refused after takeover
E [PASS] project.removal remove() drops both the directory and database; re-create afterward is genuinely empty
E [PASS] project.create_atomic an abandoned creation lock (dead pid) is healed automatically and create() completes fresh in the same call; path=/tmp/awtcontract_prepwr67/projects/contract1788426046199atomic
E [PASS] project.creation_state_reporting creation_state distinguishes none/creating/abandoned correctly
E [PASS] service.installed installed and active (unit: /home/bkrabach/.config/systemd/user/amplifier-work-tracker.service)
E [PASS] systemd.user_bus_reachable systemctl --user show-environment succeeded
E [PASS] dolt.reachable dolt sql-server responds on 127.0.0.1:35975
E [FAIL] sweeps.alive no heartbeat ever recorded for the reap sweep loop -- it may never have started, or the heartbeat file was removed; restart the service (`amplifier-work-tracker service restart`); no heartbeat ever recorded for the notify sweep loop -- it may never have started, or the heartbeat file was removed; restart the service (`amplifier-work-tracker service restart`)
E [PASS] service.restart_policy installed unit has Restart=always -- survives a clean/unintended exit
E
E 1 assumption(s) VIOLATED. Beads has changed underneath us.
E Fix scope: amplifier_work_tracker/adapter.py only -- nothing above the seam encodes Beads behaviour.
E project 'contract1788426046199atomic': healing an abandoned creation attempt (lock /tmp/awtcontract_prepwr67/projects/contract1788426046199atomic/.create.lock named a dead pid) before retrying
E
E assert 1 == 0
E + where 1 = CompletedProcess(args=['/home/bkrabach/dev/hw-model-performance/lanes/wp6-error-regex-scope/amplifier-work-tracker/.ve...(lock /tmp/awtcontract_prepwr67/projects/contract1788426046199atomic/.create.lock named a dead pid) before retrying\n").returncode

tests/cli/test_cli_surface.py:831: AssertionError
---------------------------- Captured stdout setup -----------------------------
Starting server with Config HP="127.0.0.1:35975"|T="28800000"|R="false"|L="info"
---------------------------- Captured stderr setup -----------------------------
----------------------------- Captured stdout call -----------------------------


----------------------------- Captured stderr call -----------------------------
--------------------------- Captured stderr teardown ---------------------------
=========================== short test summary info ============================
FAILED tests/cli/test_cli_surface.py::test_doctor_quick_succeeds_against_the_real_installed_bd
1 failed in 45.99s
80 changes: 80 additions & 0 deletions docs/lanes/wp6-error-regex-scope/evidence/jyg-preexisting.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
=== IS THE ONE tier-3 FAILURE PRE-EXISTING? (model_performance-jyg) ===

tests/_util.py restored to the PARENT commit ea233a7 (bare \berror\b).
The only other change on this branch is a NEW tier-1 file that this test
does not import, so this IS the parent state for this test.
command: .venv/bin/python -m pytest -m cli \
tests/cli/test_cli_surface.py::test_doctor_quick_succeeds_against_the_real_installed_bd -q

F [100%]
=================================== FAILURES ===================================
___________ test_doctor_quick_succeeds_against_the_real_installed_bd ___________

run_cli = <function run_cli.<locals>._run at 0xfd2155becea0>

def test_doctor_quick_succeeds_against_the_real_installed_bd(run_cli):
result = run_cli(["doctor", "--quick"])
> assert result.returncode == 0, result.stdout + result.stderr
E AssertionError: [PASS] version bd 1.1.2
E [PASS] capabilities all required bd commands present
E [PASS] resolve.fenced stale holder refused, as required
E [PASS] resolve.divergent_text_refused resolving a closed item with different text refuses and writes nothing
E [PASS] resolve.identical_text_idempotent re-sending identical resolution text is an idempotent success
E [PASS] reopen.reopens a resolved item reopens unassigned and is directly claimable again
E [PASS] reopen.clears_closed_at reopen clears closed_at (the documented, surfaced accounting cost)
E [PASS] reopen.close_reason_disposition reopen clears close_reason (measured), and the wrapper's archive comment preserves the previous resolution regardless
E [PASS] reopen.emits_event bd records a `reopened` events row, attributed
E [PASS] release.reopens_unresolved release reopens a held item with no resolution, and it is re-claimable
E [PASS] claim.subcommand --claim present, rejects --assignee as expected
E [PASS] claim.atomic skipped (--quick); run full doctor before trusting parallel agents
E [PASS] claim.directed_atomic skipped (--quick); run full doctor before trusting parallel agents
E [PASS] link.nonblocking discovered-from is non-blocking
E [PASS] list.includes_closed all-flag required and working
E [PASS] list.status_filter_includes_closed an explicit --status filter shows closed items without --all
E [PASS] show.dependents reverse link visible (1 links)
E [PASS] read.no_mutation repeated reads (including not-found/wrong-project misses) leave status, holder, and metadata unchanged
E [PASS] resolution.readable resolution text round-trips
E [PASS] timestamps.readable created_at/updated_at/closed_at all round-trip as real datetimes
E [PASS] metadata.roundtrip arbitrary JSON metadata round-trips
E [PASS] project.name_rules dotted names appear usable now; validator may be relaxed
E [PASS] custody.fresh_survives a fresh renewal survives regardless of total hold duration
E [PASS] custody.stale_reclaimed stale custody is reclaimed: custody stale -- last seen 3600s ago (ttl 900s)
E [PASS] custody.idle_not_exempt awaiting_human with stale custody is still reclaimed: custody stale -- last seen 3600s ago (ttl 900s)
E [PASS] custody.fenced old holder's renew and resolve are both refused after takeover
E [PASS] project.removal remove() drops both the directory and database; re-create afterward is genuinely empty
E [PASS] project.create_atomic an abandoned creation lock (dead pid) is healed automatically and create() completes fresh in the same call; path=/tmp/awtcontract_zrg7zooz/projects/contract1788425983957atomic
E [PASS] project.creation_state_reporting creation_state distinguishes none/creating/abandoned correctly
E [PASS] service.installed installed and active (unit: /home/bkrabach/.config/systemd/user/amplifier-work-tracker.service)
E [PASS] systemd.user_bus_reachable systemctl --user show-environment succeeded
E [PASS] dolt.reachable dolt sql-server responds on 127.0.0.1:45435
E [FAIL] sweeps.alive no heartbeat ever recorded for the reap sweep loop -- it may never have started, or the heartbeat file was removed; restart the service (`amplifier-work-tracker service restart`); no heartbeat ever recorded for the notify sweep loop -- it may never have started, or the heartbeat file was removed; restart the service (`amplifier-work-tracker service restart`)
E [PASS] service.restart_policy installed unit has Restart=always -- survives a clean/unintended exit
E
E 1 assumption(s) VIOLATED. Beads has changed underneath us.
E Fix scope: amplifier_work_tracker/adapter.py only -- nothing above the seam encodes Beads behaviour.
E project 'contract1788425983957atomic': healing an abandoned creation attempt (lock /tmp/awtcontract_zrg7zooz/projects/contract1788425983957atomic/.create.lock named a dead pid) before retrying
E
E assert 1 == 0
E + where 1 = CompletedProcess(args=['/home/bkrabach/dev/hw-model-performance/lanes/wp6-error-regex-scope/amplifier-work-tracker/.ve...(lock /tmp/awtcontract_zrg7zooz/projects/contract1788425983957atomic/.create.lock named a dead pid) before retrying\n").returncode

tests/cli/test_cli_surface.py:831: AssertionError
---------------------------- Captured stdout setup -----------------------------
Starting server with Config HP="127.0.0.1:45435"|T="28800000"|R="false"|L="info"
---------------------------- Captured stderr setup -----------------------------
----------------------------- Captured stdout call -----------------------------


----------------------------- Captured stderr call -----------------------------
--------------------------- Captured stderr teardown ---------------------------
=========================== short test summary info ============================
FAILED tests/cli/test_cli_surface.py::test_doctor_quick_succeeds_against_the_real_installed_bd
1 failed in 46.35s

=== CONCLUSION ===
It fails on the PARENT commit too, at the EARLIER assertion
(assert result.returncode == 0), because sweeps.alive FAILs under the
isolated AMPLIFIER_WORK_TRACKER_ROOT the run_cli fixture sets: this
machine's reap/notify heartbeats live outside that root.
assert_no_silent_failure is never reached, which is exactly why the wp6
defect is invisible locally and only appeared in CI. This is
model_performance-jyg -- separately filed, separately owned, NOT this lane's.
59 changes: 59 additions & 0 deletions docs/lanes/wp6-error-regex-scope/evidence/ledger-mutate.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
=== make ledger-mutate ===
.venv/bin/python -m ledger.checks.mutation_harness
LEDGER MUTATION HARNESS -- discriminating-power evidence
Each probe is run against a counterfactual repo and must go RED.
Injection only: no product-code edit, no subprocess, nothing written to the repo.

PINNING PROBES (row is red; probe pins the CURRENT non-conformant shape;
counterfactual = the FIXED behaviour)
(none)

CONFORMANCE PROBES (row is green; probe asserts the invariant;
counterfactual = the known-wrong shape it forbids)
[PROVEN ] CCV1-000 LEDGER-INTEGRITY (CONFORMS)
test_row_ccv1_000 :: the governed contract moved under the pinned hash
[PROVEN ] CCV1-003 REGRESSION (CONFORMS)
test_row_ccv1_003 :: the failed-custody arm returns a bare failure again (no compensating release)
[PROVEN ] CCV1-004 REGRESSION (CONFORMS)
test_row_ccv1_004 :: a failed renewal no longer stops the loop (retry returns)
[PROVEN ] CCV1-005 REGRESSION (CONFORMS)
test_row_ccv1_005 :: the custody-keeps-itself-fresh reassurance returns to the skill
[PROVEN ] CCV1-008 REGRESSION (CONFORMS)
test_row_ccv1_008 :: awareness.md states the TTL release as automatic again
[PROVEN ] CCV1-009 REGRESSION (CONFORMS)
test_row_ccv1_009 :: the close fence is re-gated on item status (the original post-reclaim gap)
[PROVEN ] CCV1-011 REGRESSION (CONFORMS)
test_row_ccv1_011 :: renewal fences on holder only -- a zombie's generation matches again
[PROVEN ] CCV1-012 REGRESSION (CONFORMS)
test_row_ccv1_012 :: release reports its outcome from the exit code, not the read-back
[PROVEN ] CCV1-013 REGRESSION (CONFORMS)
test_row_ccv1_013 :: claim_item returns an Item parsed from the writing process's own stdout
[PROVEN ] CCV1-015 REGRESSION (CONFORMS)
test_row_ccv1_015 :: _verified_write stops verifying the SUCCESS path
[PROVEN ] CCV1-016 REGRESSION (CONFORMS)
test_row_ccv1_016 :: the transaction-was-aborted guarantee Incident B disproved returns to the prose
[PROVEN ] CCV1-017 REGRESSION (CONFORMS)
test_row_ccv1_017 :: the single-hold refusal is removed from the tool seam
[PROVEN ] CCV1-021 REGRESSION (CONFORMS)
test_row_ccv1_021 :: `make test` stops collecting ledger/checks
[PROVEN ] CCV1-022 REGRESSION (CONFORMS)
test_row_ccv1_022 :: CI stops installing the tool module editable (suite runs in nothing again)
[PROVEN ] CCV1-023 REGRESSION (CONFORMS)
test_row_ccv1_023 :: FIXED: a single-hold fixture now exists in the tool module suite
[PROVEN ] CCV1-023 REGRESSION (CONFORMS)
test_row_ccv1_023 :: FIXED: the contract's Fixture 4 location line points at a real path
[PROVEN ] CCV1-023 REGRESSION (CONFORMS)
test_row_ccv1_023 :: FIXED: the contract's named fixture file tests/test_single_hold.py now exists

DENOMINATOR (this is the record)
pinning mutations proven 0 / 0
pinning probes covered proven 0 / 0
conformance mutations proven 17 / 17
ALL mutations proven 17 / 17

UNPROVEN, named with reason
(none)

Flip direction for every pinning probe above: VIOLATION-MOVEMENT.
A pinning probe going red means the behaviour moved TOWARD the contract:
update the row to CONFORMS and retarget the probe in the SAME change.
3 changes: 3 additions & 0 deletions docs/lanes/wp6-error-regex-scope/evidence/pr70-doctor-quick.run1-transient.stderr.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
project 'contract1788423815748rm': bd init hit a dirty schema migration -- dropping and retrying once: [mysql] 2026/09/03 01:25:03 connection.go:214 busy buffer
Error: failed to open Dolt store: failed to initialize schema: schema migration: pending schema migrations alter pre-existing dirty tables: comments, compaction_snapshots, dependencies, events, issue_snapshots, labels; run 'bd dolt commit' to
project 'contract1788423815748atomic': healing an abandoned creation attempt (lock /tmp/awtcontract_h8rnisef/projects/contract1788423815748atomic/.create.lock named a dead pid) before retrying
1 change: 1 addition & 0 deletions docs/lanes/wp6-error-regex-scope/evidence/pr70-doctor-quick.stderr.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
project 'contract1788424171165atomic': healing an abandoned creation attempt (lock /tmp/awtcontract_vnsevrfc/projects/contract1788424171165atomic/.create.lock named a dead pid) before retrying
Loading
Loading