[BUG] Reconcile sweep can never flag a died sidekick: the un-routed check short-circuits the liveness check

[TaekeK]

What's wrong? (one or two sentences)

The hourly reconcile sweep's died-sidekick detector (💀, added in #963) is unreachable for the issues it was written for. An earlier "no state:* label" branch in dor_reconcile.sh continues past it, and every issue in the build/feedback phase has no state:* label by design — so a dead build is reported as "the agent likely never ran" instead of "its sidekick died, re-dispatch it".

Steps to reproduce

1. Take an issue through the DoR build side until it is at functional acceptance (build-done, board Status Awaiting functional acceptance).
2. Comment feedback on it. dor_feedback_flow.sh:32 sets board Status → Building.
3. Kill the adjustment mid-flight (or let the host reclaim the sidekick, as happened twice on [Feature] Collapse managed resources under business roles in the matrix #370).
4. Wait for the hourly sweep at :17.

Real occurrence: #370 / PR #933, runs 31016109943 (5 Aug, feedback (sk3) failed 15:02:30Z) and 31076341084 (6 Aug, feedback (sk3) failed 08:41:28Z).

✅ What did you expect to happen?

The sweep flags it with the 💀 rule at dor_reconcile.sh:170-175:

💀 #370 says Building but no DoR workflow run is alive for it (Nm since its last update) — its sidekick almost certainly died mid-flight. It will not move on its own; re-dispatch it.

…and adds it to stalled, which drives the direct comment on the issue at :215.

❌ What actually happened?

The sweep saw Status: Building and still emitted the un-routed message. From the DoR pipeline health issue #886, sweep of 2026-08-06 09:09 UTC:

🕳️ #370 is on the board (Status: Building) with no state:* label after 1151h — the agent likely never ran.

Nothing was flagged as dead, no comment landed on #370, and no sidekick was re-dispatched. The work sat dead for ~18 hours (5 Aug 15:02 → 6 Aug 09:27) across roughly 18 sweeps, until @WimvandenHeijkant noticed by hand and re-posted both rounds of requestor feedback himself.

Root cause

.github/scripts/dor_reconcile.sh:143-150:

if [ -z "$state_label" ]; then
  age_h=$(( (now - created_epoch) / 3600 ))
  if [ "$age_h" -ge "$UNROUTED_HOURS" ]; then
    add_ex "🕳️ #${num} ... with no \`state:*\` label after ${age_h}h — the agent likely never ran."
  fi
  continue        # <-- skips the rest of the loop body
fi

That continue skips everything below it, including the "Building") liveness case at :170 and the zombie-sidekick check at :183.

An issue on the build side has no state:* label by design — the build phase is tracked with build-done / ready-to-build, and dor_build_flow.sh removes state:awaiting-approval when the build starts. #370's labels are enhancement, build-done, sk:sk3. So the whole build/feedback population takes the un-routed branch, which is precisely the population the 💀 rule exists for.

Two aggravating details:

• The un-routed message measures age from issue creation (created_epoch), so a build that died 18 hours ago is reported as "after 1151h" ([Feature] Collapse managed resources under business roles in the matrix #370 was opened 19 Jun). It reads as ancient backlog noise, not as something that just broke.
• The wrong branch also produces the wrong instruction. "The agent likely never ran" invites no action; "re-dispatch it" is what was actually needed.

Suggested direction

Don't let un-routed short-circuit liveness. Either run the Building liveness check before the state_label test, or drop the continue and make the un-routed flag skip only the drift/stale checks that genuinely need a label. Worth also measuring the un-routed age from updated_epoch, or suppressing it entirely while Status is an active phase — an issue mid-build is not un-routed.

Does it reproduce on the demo data?

Not applicable — this is CI/pipeline automation, not product code. It reproduces against the repo's own Actions history (see the two run IDs above and health issue #886).

Where does it happen? (optional)

.github/scripts/dor_reconcile.sh (dor-reconcile.yml, hourly at :17).

Did it work before? (optional)

No — the 💀 rule has been shadowed since it was introduced in #963. Its own comment says "Three builds died this way in a single day and every one of them was found by a human watching, not by this sweep"; that is still true, and this is why.

[github-actions]

@TaekeK — I traced every claim in the report against the current scripts, and it is correct in every particular: the 💀 died-sidekick rule has been unreachable for its target population since it shipped in #963. That said, this is a defect in the DoR automation itself (.github/scripts/dor_reconcile.sh + dor-reconcile.yml), not product code — so per the process and the standing precedent for [CI] issues (#801, #800, #721, #716), it routes out of pipeline and gets fixed through the normal dev flow. Everything a fixer needs is consolidated below.

Verification (static trace against the real code)

1. The short-circuit is real. .github/scripts/dor_reconcile.sh:143-150: when state_label is empty, the loop flags 🕳️ un-routed (if ≥ UNROUTED_HOURS) and continues at :149 — skipping the drift check (:154-157), the entire case "$status" block (:160-177) including the "Building" 💀 liveness arm (:170-176), and the zombie-sidekick check (:183-187).
2. The build/feedback population always takes that branch. dor_build_flow.sh:214 removes state:awaiting-approval when build-done is applied, and nothing re-adds a state:* label afterwards; dor_feedback_flow.sh:32 sets board Status → Building when feedback arrives. So an issue in exactly the state the 💀 rule describes — Status Building, sidekick dead — always has an empty state_label and can never reach :170.
3. The per-issue alarm is equally dead. stalled is only populated at :174, so the dor-stuck label + direct comment machinery at :212-216 — the only signal that reaches the issue itself — is unreachable for the same population.
4. Both pipelines are affected. dor-reconcile.yml runs the script twice: default LABEL=enhancement and again with LABEL=bug, so a died bug-fix build is invisible the same way.
5. Aggravations confirmed. The un-routed message measures age from created_epoch (:145) — hence "[Feature] Collapse managed resources under business roles in the matrix #370 … after 1151h" for a build that died 18 hours earlier — and its text ("the agent likely never ran") prescribes the wrong action.

I can't replay the Actions runs from this environment, but the static trace is unambiguous and your #370 / #886 timeline (runs 31016109943, 31076341084) matches exactly what this control flow must produce. Confirmed: real bug, and the 💀 rule has been shadowed since introduction.

Root cause

Control-flow ordering in dor_reconcile.sh, spec-side automation layer: the un-routed check conflates "no state:* label" with "never entered the pipeline", when no-state-label is also the designed labelling of the whole build/feedback phase — and the continue at :149 makes un-routed mutually exclusive with every check below it. One conflation, one early exit.

Fix sketch + blast radius (for whoever picks it up)

• Evaluate liveness regardless of the state label: move the case "$status" block (or at minimum the "Building" arm) above the [ -z "$state_label" ] test, or drop the continue and guard only the checks that genuinely need a label (drift :154-157, stale-waiting :161-164).
• Scope 🕳️ un-routed to genuinely un-routed issues — Status empty/backlog-ish; an issue whose Status is an active phase is by definition routed. Measure its age from updated_epoch, not created_epoch.
• Regression-risk notes: keep 🕳️ firing for the true never-triaged case (on board, no label, no meaningful Status); the needs-vouch (:126-130) and heal (:132-141) branches must stay ahead of everything; and the :212-216 block comments on issues and @-mentions real people, so preserve the fail-safe direction of has_live_run (:90-96 — unknown ⇒ alive) to avoid false "your build is dead" spam. The zombie check (:183-187) becoming reachable for label-less issues is intended behaviour, but worth a moment's thought for label-less non-build issues.
• Only consumer of the script is dor-reconcile.yml (2 invocations); output lands in the two health issues (DoR pipeline health #886, DoR bug-pipeline health #938) and, via 3c, on stalled issues directly. No product code involved.

Regression test

There is no harness for .github/scripts/*.sh today — part of why this shipped shadowed. A minimal red test (e.g. bats or a plain bash assert script) would stub gh on PATH to serve: a board snapshot with issue N at Status Building, an issue list row with state_label="" + sk:skX + updated_epoch 30+ min old, and an empty live-runs list — then assert the output contains the 💀 line for N and that N is marked dor-stuck. Today it emits 🕳️ (or nothing, inside UNROUTED_HOURS) instead. The coverage ratchet exempts CI files, so this needs a small new harness; recommended while touching the file, but the fixer's call.

Duplicates / related

No duplicate in the open backlog. #886 / #938 are the health issues where the symptom surfaces, not reports of it; #801 / #800 are the adjacent out-of-pipeline CI-hardening umbrellas if this wants a home.

Route: state:out-of-pipeline — confirmed and root-caused, but CI/tooling bugs are handled by the normal dev flow, not the autonomous build pipeline.