fix(dor): un-shadow the died-sidekick check in the reconcile sweep - #996
Merged
Conversation
Closes #995 The 💀 liveness rule added in #963 has been unreachable for its entire target population since it shipped. `dor_reconcile.sh:143-150` flagged "no `state:*` label" as un-routed and `continue`d — skipping the drift check, the whole `case "$status"` block including the `"Building"` liveness arm, and the zombie sidekick check. But "no state label" is not un-routed: the build side deliberately runs without one. dor_build_flow.sh drops `state:awaiting-approval` when it applies `build-done`, nothing re-adds a state label, and dor_feedback_flow.sh sets board Status to Building on feedback. So an issue in exactly the state the rule describes — Status Building, sidekick dead — always arrived here label-less and could never reach the check written for it. `stalled` was only ever populated inside that arm, so the `dor-stuck` label and the direct comment on the issue, the only signal that reaches a human, were dead too. Both pipelines: the workflow runs the script for enhancement and again for bug. Observed on #370: runs 31016109943 and 31076341084 died when their sidekick was reclaimed; the sweep reported "🕳️ #370 is on the board (Status: Building) with no state:* label after 1151h — the agent likely never ran" (health issue #886, 6 Aug 09:09). It had Status=Building in hand and still took the un-routed branch. The work sat dead ~18h across ~18 sweeps until a human re-posted the requestor's feedback by hand. Ask the board whether anything owns the issue instead of trusting the label: a new `build_phase()` covers the four Statuses the build side owns, and a label-less issue in one of them now falls through to the liveness and zombie checks. Genuinely un-routed issues still flag as before; the drift check is explicitly label-guarded. Un-routed age now comes from updated_epoch, not created_epoch — that "1151h" was measured from June, for something that had been dead 18 hours. test/ci-scripts/test-dor-reconcile-liveness.sh is a first harness for .github/scripts/*.sh — the absence of one is why this shipped shadowed. It puts a stub `gh` on PATH and drives the real script end to end. 9 assertions; 7 fail against the unfixed script, including the exact #995 shape (dead build reported 🕳️ instead of 💀). The two that pass pre-fix are the ones that should: a live build stays quiet, and a genuinely un-routed issue is still flagged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #995
The 💀 liveness rule added in #963 has been unreachable for its entire target population since it shipped.
The bug
dor_reconcile.sh:143-150flagged "nostate:*label" as un-routed andcontinued — skipping the drift check, the wholecase "$status"block including the"Building"liveness arm (:170), and the zombie-sidekick check (:183).But "no state label" is not un-routed. The build side deliberately runs without one:
dor_build_flow.shdropsstate:awaiting-approvalwhen it appliesbuild-done, nothing re-adds a state label, anddor_feedback_flow.sh:32sets board Status to Building on feedback. So an issue in exactly the state the rule describes — Status Building, sidekick dead — always arrived here label-less and could never reach the check written for it.stalledis only populated inside that arm, so thedor-stucklabel and the direct comment at:240-244— the only signal that reaches a human — were dead for the same reason. Both pipelines are affected:dor-reconcile.ymlruns the script forenhancementand again forbug.Observed
On #370, runs
31016109943and31076341084died when their sidekick was reclaimed. The sweep reported (health issue #886, 6 Aug 09:09):It had
Status: Buildingin hand and still took the un-routed branch. The work sat dead ~18 hours across ~18 sweeps until @WimvandenHeijkant noticed by hand and re-posted both rounds of the requestor's feedback himself.The fix
Ask the board whether anything owns the issue rather than trusting the label. A new
build_phase()covers the four Statuses the build side owns (Building,Awaiting functional acceptance,Awaiting merge,Exceptions), and a label-less issue in one of them now falls through to the liveness and zombie checks.updated_epoch, notcreated_epoch. That "1151h" was measured from June, for something that had been dead 18 hours — it read as ancient backlog rather than something that had just broken.Tests
test/ci-scripts/test-dor-reconcile-liveness.sh— a first harness for.github/scripts/*.sh. The absence of one is why this shipped shadowed and stayed that way for weeks. It puts a stubghonPATHserving fixtures, drives the real script end to end, and asserts on the health report it publishes plus the writes it makes. No network, no tokens.9 assertions. 7 fail against the unfixed script, including the exact #995 shape:
The two that pass pre-fix are the ones that should, which is what shows the fix is narrow: a live build stays quiet, and a genuinely un-routed issue is still flagged.
Wired into the existing
ci-scriptsjob inpr.yml, alongside the guards added in #994.Note on how this was routed
The DoR bug agent independently traced #995 against the current scripts, confirmed it "correct in every particular", and routed it
state:out-of-pipeline— correct, since CI/automation defects go through the normal dev flow rather than the autonomous builder. Its fix sketch and regression-risk notes shaped this change; thebuild_phase()approach is the first of the two options it laid out.🤖 Generated with Claude Code