probe: which branch decides liveness (#954 diagnostics, NOT FOR MERGE) - #992
probe: which branch decides liveness (#954 diagnostics, NOT FOR MERGE)#992fujibee wants to merge 2 commits into
Conversation
…ERGE Diagnostic only, for #970 / #954. This branch exists to produce one CI run and be deleted. `_agmsg_pid_alive_local` gained two verdicts that used to be "dead": ps ran, pid absent, canary absent dead -> alive ps failed entirely dead -> alive Both open in the same direction — unknown becomes alive — and two macOS shards fail on this PR in ways that would follow from a liveness answer that never says dead. Whether they actually enter those branches on the runner is not something I can tell from here, and this stops me guessing. Appends to a file rather than writing stderr. The helper has 61 call sites, some of which capture stderr and some of whose tests assert on it; a probe that changes what a caller sees is measuring itself. Off unless AGMSG_PIDALIVE_TRACE is set. One trace point had to be removed after it broke a test: `instance-id: liveness answers alive on the builtin, before any subshell` reads the function's own text and requires the fast path to end in `return 0;`. Wrapping that line in braces for the probe changed the shape it asserts. That is the hazard this file's conditions were about, met in the first attempt. Measured, same suite, probe off and on: 67/67 both ways, 17 trace lines written. Locally every UNKNOWN comes from #954's own tests, which break `ps` on purpose — the ordinary paths do not reach it here.
…FOR MERGE Enables AGMSG_PIDALIVE_TRACE for the shard run and uploads the file. always() on the upload, because a GREEN shard's distribution is the control: reading only the red ones cannot separate 'entered that branch and therefore failed' from 'enters that branch every time'. Writes to a file, never to the step's stdout — the same reason the hang sampler beside it does, and the reason the helper's own probe does: an instrument on a channel some callers capture would be measuring itself. Step shape and indentation checked against the existing 'Upload hang samples' step, since no YAML parser is available locally.
|
Closing without merging — the measurement is no longer worth taking. This branch was going to record which liveness branch the runner entered. Reading the withdrawal document on #693 (comment 5239715879) made the answer irrelevant either way:
The document puts it directly: "the walk produces only negative evidence, so what is needed is positive evidence." #970 changed which default an unusable observation resolves to. This probe could only have told me which default was reached — a question on the same axis, when the axis itself is what needs replacing. #737 tried the other end of that axis and was withdrawn for it: absence of a visible ancestor read as death, which killed live watchers. #970 reads a failed observation as life, which leaves dead watchers running. Same axis, both ends broken. The probe design is kept here in case a later measurement needs it: file-append rather than stderr (61 call sites, some capturing it), env-gated so an ordinary run is untouched, uploaded on |
Do not merge. This branch exists to produce one CI run and then be deleted.
Why
#970(the #954 fix) turns two liveness verdicts from dead into alive:psran, pid absent, canary absentpsfailed entirelyBoth open in the same direction — unknown becomes alive — and two macOS shards fail on that PR in ways that would follow from a liveness answer that can no longer say dead:
watch: exits when its session dies without consuming an undelivered row— fails onrun kill -0 "$w", i.e. the watcher did not exitsync start: starts a stopped engine and status reports it runningNeither reproduces here: 12/12 and 31/31 locally. So this records which branch actually decided, on the runner, instead of me guessing from the outside.
How it is kept from changing what it measures
AGMSG_PIDALIVE_TRACEis set, so an ordinary run is untouched.always()— a green shard's distribution is the control. Reading only red shards cannot separate "entered that branch and therefore failed" from "enters that branch every time".Measured, same suite with the probe off and on: 67/67 both ways.
One trace point had to be removed
instance-id: liveness answers alive on the builtin, before any subshellreads the function's own text and requires the fast path to end inreturn 0;. Wrapping that line in braces for the probe changed the shape it asserts, and the test went red — the exact hazard this design was meant to avoid, met on the first attempt. That branch is now untraced.Locally
Every
UNKNOWN-assumed-alivecomes from #954's own tests, which breakpson purpose. The ordinary paths do not reach it on this machine — which is the whole reason to look at the runner.