fix(cairn tests): name the I/O stall, so a 5s fsync stops reading as a code failure

[ZacxDev]

The failure

tekton/devrc-pytests fails on test_cairn_write.py::TestAppendLands::test_a_bullet_is_appended_and_the_status_is_named on PRs whose diff cannot reach it:

AssertionError: 🔴 cairn: the write did NOT happen — http://127.0.0.1:PORT unreachable: timed out
                Nothing was queued and nothing was written locally.
assert 7 == 0

That is a sentence about the write client describing what was a disk stall.

The mechanism — read off the CI log, not inferred by analogy

🔴 The store-api root cause was NOT assumed to transfer. test_cairn_write.py imports http.server, stands up its own loopback servers, imports no store-api server.py, and carried no MECHANISM classifier — all verified first.

Established from devrc-ci-jfg67 directly:

• the store root is /tmp/nix-build-devrc-pytests.drv-0/pytest-of-nixbld1/pytest-0/popen-gw1/… — the step container's ephemeral layer, the same device the store-api half stalls on;
• the client reports timed out, not a refused connection;
• server.py:_replace_bytes fsyncs the file and then the parent directory INSIDE the request, before the response is written;
• run_cairn passes --timeout 5.

So one fsync slower than five seconds is a refusal at exit 7. 🔴 That bound is twelve times tighter than the store-api half's HANG_TIMEOUT of 60.0, which is why this file is the more frequent casualty of the same node contention.

All 5 failures in that run were exactly the 5 write-success assertions — 4× TestAppendLands plus test_put_derives_the_revision_from_a_LIVE_sync_and_the_replace_lands (--dist loadfile puts the file on one worker). Those are precisely the 5 sites this PR instruments.

What this changes: the message, and nothing else

🔴 DIAGNOSIS, NOT TOLERANCE. No bound moved, nothing retries, no test was made to pass. A gate that reports a code failure for an I/O stall trains everyone to click through — that was the whole cost.

• scripts/testlib/hang_mechanism.py (new) — a MECHANISM = verdict from the live thread stacks, the shape test_subsystem_store_api.py already had. The failure now reads:

  MECHANISM = SERVER_BLOCKED_IN_FSYNC   (handler threads=2 [...=BLOCKED_ELSEWHERE
  ...=SERVER_BLOCKED_IN_FSYNC], accept loop parked=True)
  store=/dev/shm/devrc-store-… fs=tmpfs
  

• 🔴 The headline is deliberately NOT a consensus of the handler threads. Two servers are live here — the store and the shim in front of it — so on a timeout they legitimately disagree (shim in urlopen, store in fsync). Measured, not anticipated. A rule requiring agreement would answer AMBIGUOUS for the textbook case the classifier exists to name.
• 🔴 It scans frames' SOURCE LINES, never their FILENAME — the defect _HUNG_SERVER_RULES carries as known-and-unfixed, where a worktree named devrc-fsync misclassifies every hang. test_subsystem_store_api.py was not rewired onto the shared module: it has its own tests and that is its own edit. Two copies exist today; that is known debt, not an oversight.
• scripts/ci-repro/slow_cairn_fsync.py (new) — stalls os.fsync in the test process. Not LD_PRELOAD: the server is in-process while the client is a subprocess, so preloading would stall both sides and muddy which one timed out.

Evidence

Reproduced on the dev host, text identical to CI:

run	result
control (inert)	4 passed in 3.69s, intercepted_fsyncs=8
armed SLOW_CAIRN_FSYNC_S=8	4 failed in 28.95s

intercepted_fsyncs=8 is the positive control — the append path really does issue the two _replace_bytes fsyncs per write; a zero would mean the shim patched nothing. Both selftest arms watched to fire (inert-run abort; zero-fsync abort on a 2 passed selection).

⚠ The store was on tmpfs for BOTH runs (/dev/shm/devrc-store-…), i.e. with store_siting's mitigation fully in force. This is a LATENCY dependency, not a filesystem one. tmpfs makes a breach far less likely; it does not remove the 5 s bound.

Mutation matrix on the new classifier — every mutant killed, each by the right test

mutant	killed by
filename folded back into the scanned text	path test, with its own message
fsync rule removed from RULES	5 tests
consensus rule (disagreement ⇒ no verdict)	headline test
classify always answers fsync	3 tests, incl. the pull-apart
outermost frame wins	innermost test

Run under PYTHONDONTWRITEBYTECODE=1; file restored byte-identical to its pre-mutation backup afterwards.

What is NOT fixed here, and is not claimed

• The stall itself. It is node-local device contention; the levers are infra and ranked in the ci-repro README.
• The disk-siting half was already fixed by b4fde334 (fix(store tests): one shared store siting + a ledger, because #1211 reached one of three sites #1219).
• ⚠ A PR branched before b4fde334 still carries the old disk-backed fixture — branch protection sets strict: false, so nothing rebases it. That is why feat(handoff-search): section-grained full-text index of the handoff corpus (P1) #1209 and fix(subsystem-store): repoint the host-local READ path off the frozen mirror, and refuse an undateable store #1233 were still failing; rebasing is what picks the mitigation up.
• Measured, and it is good news: store_siting is genuinely live in CI, not silently falling back. devrc-ci-826jc ran rev 946a51f0 (contains the fix and the siting tests) with failed=0 and only 3 skips (signal×2, repo-cos) — so test_the_store_fixture_ACTUALLY_lands_on_tmpfs_when_one_exists, which skips when no tmpfs is usable, ran and passed.

Gate

Verdicts in the PR comments below (dev-host tier + both nix check derivations, built one at a time).

[ZacxDev]

Gate verdicts

Rebased onto 5a82aaa9 (current main) before these runs, so the PR is not gated on a stale tree.

Sandbox tier — the two nix check derivations, built one at a time

derivation	verdict
.#checks.x86_64-linux.nodetests	RESULT: PASS (exit=0) — TOTAL suites=5 files=41 tests=1449 pass=1449 fail=0 skipped=0 (floor: 1367), output path /nix/store/9qdvf84s3nczap7cy2ma4s6zklf8bp11-devrc-nodetests valid
.#checks.x86_64-linux.pytests	RESULT: FAIL (exit=1) — 1 failure, and it is PRE-EXISTING ON main (below)

🔴 The one pytests failure is not this PR, and that is proved, not asserted.

scripts/collector/keylog/tests/test_espanso_detect.py::test_live_existing_resolutions_not_made_ambiguous
AssertionError: search terms regressed:
  {'recom': (':rna', None, [':acq', ':rna']), 'recommend': (':rna', None, [':acq', ':rna'])}

Cause: a720d30d ("espanso"), one of the five commits already on main, changed the :acq
snippet's label from "ask clarifying questions" to
"ask clarifying questions and recommend improvements and anything useful to include". The word
"recommend" now makes recom/recommend match :acq as well as :rna, so _attribute
returns None and the pinned resolution regresses.

Discriminating control — the file that decides it is byte-identical between this branch and
origin/main (git diff origin/main -- nix/home.nix is empty), and reverting only that file
flips the result:

nix/home.nix from	result
a720d30d^	1 passed
a720d30d (= current main = this branch)	1 failed

So origin/main is red on the sandbox pytests tier right now, independently of this PR. This
diff touches no espanso config and cannot reach that test. Fixing it is a judgement call I
deliberately did not guess at — either the new :acq label is intended and the pinned
recom -> :rna resolution needs updating, or the label should not carry "recommend". That is the
config owner's call.

Dev-host tier — scripts/gate.sh --tier both

Run before the rebase (base 946a51f0):

tier	verdict
node	RESULT: PASS (exit=0) — tests=1449 pass=1449 fail=0
pytest	RESULT: FAIL (exit=1) — 1 failed, 11502 passed in 612.89s; TOTAL collected=20500 passed=20496 skipped=3 failed=1

That single failure was test_git_repo_isolation.py::test_live_cotenants_sees_another_process_in_the_repo:

assert live_cotenants([git_dir]) == [], "a brand-new tmp repo already has tenants?"
AssertionError: assert ['522745:git'] == []

A stray git process was inside a brand-new tmp repo. The same log records a concurrent session
pushing another branch during the run (refs/heads/feat/handoff-search-index-p1
926b82ce -> 0906bbb6), which is where that process came from — this box runs parallel agents.
Re-run of the whole file: 114 passed. Transient, and this diff cannot reach live_cotenants.

Change-specific verification

• test_cairn_write.py + test_hang_mechanism.py + test_store_siting_ledger.py: 46 passed.
• Mutation sweep on the new classifier: 5 mutants, 5 killed, each by the intended test, run under
  PYTHONDONTWRITEBYTECODE=1; module restored byte-identical to its pre-mutation backup.
• Reproduction instrument validated in both directions, with both selftest arms watched to fire.

⚠ Honest scope: I do not have a fully green sandbox pytests verdict for this tree, because
main itself is red. What I have is that the tier's only failure is attributed, reproduced, and
flipped by reverting a file this PR does not touch.