Skip to content

probe: which branch decides liveness (#954 diagnostics, NOT FOR MERGE) - #992

Closed
fujibee wants to merge 2 commits into
mainfrom
probe/954-branch-trace
Closed

probe: which branch decides liveness (#954 diagnostics, NOT FOR MERGE)#992
fujibee wants to merge 2 commits into
mainfrom
probe/954-branch-trace

Conversation

@fujibee

@fujibee fujibee commented Aug 25, 2026

Copy link
Copy Markdown
Owner

Do not merge. This branch exists to produce one CI run and then be deleted.

Why

#970 (the #954 fix) turns two liveness verdicts from dead into alive:

situation before after
ps ran, pid absent, canary absent dead alive
ps failed entirely dead alive

Both open in the same direction — unknown becomes alive — and two macOS shards fail on that PR in ways that would follow from a liveness answer that can no longer say dead:

  • watch: exits when its session dies without consuming an undelivered row — fails on run kill -0 "$w", i.e. the watcher did not exit
  • sync start: starts a stopped engine and status reports it running

Neither reproduces here: 12/12 and 31/31 locally. So this records which branch actually decided, on the runner, instead of me guessing from the outside.

How it is kept from changing what it measures

  • Appends to a file, never stderr. The helper has 61 call sites; some capture stderr, some have tests asserting on it. A probe on that channel would be measuring itself.
  • Off unless AGMSG_PIDALIVE_TRACE is set, so an ordinary run is untouched.
  • Uploaded on always() — a green shard's distribution is the control. Reading only red shards cannot separate "entered that branch and therefore failed" from "enters that branch every time".

Measured, same suite with the probe off and on: 67/67 both ways.

One trace point had to be removed

instance-id: liveness answers alive on the builtin, before any subshell reads the function's own text and requires the fast path to end in return 0;. Wrapping that line in braces for the probe changed the shape it asserts, and the test went red — the exact hazard this design was meant to avoid, met on the first attempt. That branch is now untraced.

Locally

Every UNKNOWN-assumed-alive comes from #954's own tests, which break ps on purpose. The ordinary paths do not reach it on this machine — which is the whole reason to look at the runner.

…ERGE

Diagnostic only, for #970 / #954. This branch exists to produce one CI run and
be deleted.

`_agmsg_pid_alive_local` gained two verdicts that used to be "dead":

  ps ran, pid absent, canary absent   dead -> alive
  ps failed entirely                  dead -> alive

Both open in the same direction — unknown becomes alive — and two macOS shards
fail on this PR in ways that would follow from a liveness answer that never says
dead. Whether they actually enter those branches on the runner is not something
I can tell from here, and this stops me guessing.

Appends to a file rather than writing stderr. The helper has 61 call sites, some
of which capture stderr and some of whose tests assert on it; a probe that
changes what a caller sees is measuring itself.

Off unless AGMSG_PIDALIVE_TRACE is set.

One trace point had to be removed after it broke a test:
`instance-id: liveness answers alive on the builtin, before any subshell` reads
the function's own text and requires the fast path to end in `return 0;`.
Wrapping that line in braces for the probe changed the shape it asserts. That is
the hazard this file's conditions were about, met in the first attempt.

Measured, same suite, probe off and on: 67/67 both ways, 17 trace lines written.
Locally every UNKNOWN comes from #954's own tests, which break `ps` on purpose —
the ordinary paths do not reach it here.
…FOR MERGE

Enables AGMSG_PIDALIVE_TRACE for the shard run and uploads the file.

always() on the upload, because a GREEN shard's distribution is the control: reading only the red ones cannot separate 'entered that branch and therefore failed' from 'enters that branch every time'.

Writes to a file, never to the step's stdout — the same reason the hang sampler beside it does, and the reason the helper's own probe does: an instrument on a channel some callers capture would be measuring itself.

Step shape and indentation checked against the existing 'Upload hang samples' step, since no YAML parser is available locally.
@fujibee

fujibee commented Aug 25, 2026

Copy link
Copy Markdown
Owner Author

Closing without merging — the measurement is no longer worth taking.

This branch was going to record which liveness branch the runner entered. Reading the withdrawal document on #693 (comment 5239715879) made the answer irrelevant either way:

The document puts it directly: "the walk produces only negative evidence, so what is needed is positive evidence." #970 changed which default an unusable observation resolves to. This probe could only have told me which default was reached — a question on the same axis, when the axis itself is what needs replacing.

#737 tried the other end of that axis and was withdrawn for it: absence of a visible ancestor read as death, which killed live watchers. #970 reads a failed observation as life, which leaves dead watchers running. Same axis, both ends broken.

The probe design is kept here in case a later measurement needs it: file-append rather than stderr (61 call sites, some capturing it), env-gated so an ordinary run is untouched, uploaded on always() so a green shard is the control. One trace point had to be removed after it broke instance-id: liveness answers alive on the builtin, before any subshell, which asserts on the function's own text — worth knowing before instrumenting that helper again.

@fujibee fujibee closed this Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant