Repository navigation
đź”´ Estate CI-red master cause: @d135b05 reusable bump broke ~300 repos (diagnosis + remediation plan)Â #426
Copy link
Copy link
Closed
Description
Round-2 estate CI audit (2026-06-25): ~300 of 377 repos have red CI; ~207 have required-check deadlock (failing shared checks required for merge — the reason ~everything has needed admin-merge). Root-caused from live failed-run logs. No changes made yet (owner chose diagnose-first).
MASTER CAUSE
The 2026-06-23/24 standardization sweep that bumped all wrapper pins to @d135b05 introduced two startup-failure bugs in the shared reusables/wrappers:
- Bug A —
timeout-minutes:on auses:reusable-caller job. GitHub forbids it: "when a reusable workflow is called with 'uses', 'timeout-minutes' is not available." → run fails validation, 0 jobs. Ingovernance.yml,secret-scanner.yml,mirror.ymlwrappers (generator-emitted). - Bug B — reusable job requests permissions above the caller's cap. At
@d135b05the scorecard + secret-scanner(gitleaks) reusables gained job-levelpermissions:requestingsecurity-events: write/id-token: write/pull-requests: write; callers cap at workflow-levelcontents: read. GitHub: "permissions can only be maintained or reduced — not elevated." → unsatisfiable → startup_failure. Proven by the sharp transition (success at@3e4bd4c→ startup_failure at@d135b05, wrapper otherwise identical).
Per-workflow root cause
| Workflow | ~count | Type | Root cause | Fix |
|---|---|---|---|---|
| OSSF Scorecard | 90 | startup_failure | Bug B (security-events/id-token writes exceed cap) | declare writes at reusable top-level permissions: (or grant at caller workflow-level); re-pin |
Secret Scanner (scan/*) |
87 | startup_failure | Bug B (gitleaks job perms) + Bug A (timeout on some wrappers) | remove gitleaks job-perms block / hoist scopes; strip timeout-minutes; re-pin |
| Governance | 48 | startup_failure | Bug A | delete timeout-minutes: from governance.yml uses: job + fix generator |
| Hypatia Scan | 47 | step-failure | hypatia-cli.sh scan . exits 1 on ≥medium findings as first line of a bash -e step → aborts before the warn-don't-fail logic (all critical=0) |
append ` |
| dogfood-gate (K9) | 52 | step-failure | validator drift: k9-validate-action@2d96f43c now requires literal K9! first line + signature_required; templates never migrated |
fix *.k9.ncl templates OR relax the validator to accept a leading SPDX comment before K9! |
| mirror.yml | 56 | mixed | Bug A (timeout) + stale pre-#416 reds + skipped-misread-as-failure |
strip timeout-minutes; the rest is expected-noise |
Remediation plan (highest leverage first)
- Strip
timeout-minutes:from every reusable-caller (uses:) job (Bug A) + fix the wrapper generator so it stops emitting it. Detector:actionlint. → clears Governance + much of Secret Scanner + Mirror. - Fix Bug B: declare needed write scopes at the reusable top-level
permissions:inscorecard-reusable.yml+secret-scanner-reusable.yml(and/or caller workflow-level), then re-pin estate-wide. → clears Scorecard + clean-wrapper Secret Scanner. || trueon the hypatia scan line. → clears ~47.- Drop stale required contexts (e.g.
scan / trufflehog, renamed) from each ruleset that still lists them (owner-gated, per-repo) → clears the masked deadlock on the ~207. - K9 template migration (owner-gated) → dogfood-gate.
Items 1–3 are single central edits (reusables + generator) that clear the large majority of the ~300 red repos; 4–5 are owner-gated/per-repo. After 1–3, re-run the audit and reassess the deadlock before touching rulesets.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
Estate audit & optimization — Wave 0: kill the false green (holes bef…
Estate audit — Wave 3: MUST/SHOULD/COULD compliance scorecards + gene…
Estate audit — Wave 4: DYADT, post-action agent-claim verification (r…
hyperpolymath commentedon Aug 7, 2026
Re-measured 2026-08-07 — still live, and the remediation has barely moved.
This issue diagnoses
@d135b05as the master cause of estate CI-red, affecting~300 repos. Checking whether that pin has actually been rolled forward:
270 files across the org still reference that exact SHA. The pin that this
issue identifies as the master cause is still the pin in force nearly everywhere.
Two things follow:
remediation is what has not happened.
2026-06-25 with no recorded movement in either direction.
That second point is the reason for #591 — a
Debtfileentry with a probe and a ceiling, so a number like this gets re-checked weekly
instead of believed. A suitable entry:
fix(ci): remove timeout-minutes from job using reusable workflow
fix(ci): remove timeout-minutes from job using reusable workflow
fix(ci): remove timeout-minutes from job using reusable workflow
fix(ci): update reusable workflow pins to @892497fe + remove excessiv…
fix(ci): update reusable workflow pins to @892497fe + remove excessiv…
fix(ci): update reusable workflow pins to @892497fe + remove excessiv…
fix(ci): update reusable workflow pins to @892497fe + remove excessiv…
fix(ci): update reusable workflow pins to @892497fe + remove excessiv…
fix(ci): update reusable workflow pins to @892497fe + remove excessiv…
fix(ci): update reusable workflow pins to @892497fe + remove excessiv…
fix(ci): update reusable workflow pins to @892497fe + remove guix.scm…
128 remaining items
chore(ci): bump standards reusable pins to 5b1d0022 (#426)
fix(ci): add required permissions for reusable workflows (Bug B)
chore(ci): bump standards reusable pins to fix Bug A and Bug B (#426)
chore(ci): bump standards reusable pins to 5b1d0022 (#426)
fix(ci): add required permissions for reusable workflows (Bug B)
fix(ci): add required permissions for reusable workflows (Bug B)
chore(ci): bump standards reusable pins to fix Bug A and Bug B (#426)
chore(ci): bump standards reusable pins to 5b1d0022 (#426)
fix(ci): add required permissions for reusable workflows (Bug B)
chore(ci): bump standards reusable pins to fix Bug A and Bug B (#426)
chore(ci): bump standards reusable pins to 5b1d0022 (#426)
fix(ci): add required permissions for reusable workflows (Bug B)
chore(ci): bump standards reusable pins to fix Bug A and Bug B (#426)
chore(ci): bump standards reusable pins to 5b1d0022 (#426)
fix(ci): add required permissions for reusable workflows (Bug B)