Skip to content

🔴 Estate CI-red master cause: @d135b05 reusable bump broke ~300 repos (diagnosis + remediation plan) #426

Closed

Description

@hyperpolymath

Round-2 estate CI audit (2026-06-25): ~300 of 377 repos have red CI; ~207 have required-check deadlock (failing shared checks required for merge — the reason ~everything has needed admin-merge). Root-caused from live failed-run logs. No changes made yet (owner chose diagnose-first).

MASTER CAUSE

The 2026-06-23/24 standardization sweep that bumped all wrapper pins to @d135b05 introduced two startup-failure bugs in the shared reusables/wrappers:

  • Bug A — timeout-minutes: on a uses: reusable-caller job. GitHub forbids it: "when a reusable workflow is called with 'uses', 'timeout-minutes' is not available." → run fails validation, 0 jobs. In governance.yml, secret-scanner.yml, mirror.yml wrappers (generator-emitted).
  • Bug B — reusable job requests permissions above the caller's cap. At @d135b05 the scorecard + secret-scanner(gitleaks) reusables gained job-level permissions: requesting security-events: write / id-token: write / pull-requests: write; callers cap at workflow-level contents: read. GitHub: "permissions can only be maintained or reduced — not elevated." → unsatisfiable → startup_failure. Proven by the sharp transition (success at @3e4bd4c → startup_failure at @d135b05, wrapper otherwise identical).

Per-workflow root cause

Workflow ~count Type Root cause Fix
OSSF Scorecard 90 startup_failure Bug B (security-events/id-token writes exceed cap) declare writes at reusable top-level permissions: (or grant at caller workflow-level); re-pin
Secret Scanner (scan/*) 87 startup_failure Bug B (gitleaks job perms) + Bug A (timeout on some wrappers) remove gitleaks job-perms block / hoist scopes; strip timeout-minutes; re-pin
Governance 48 startup_failure Bug A delete timeout-minutes: from governance.yml uses: job + fix generator
Hypatia Scan 47 step-failure hypatia-cli.sh scan . exits 1 on ≥medium findings as first line of a bash -e step → aborts before the warn-don't-fail logic (all critical=0) append `
dogfood-gate (K9) 52 step-failure validator drift: k9-validate-action@2d96f43c now requires literal K9! first line + signature_required; templates never migrated fix *.k9.ncl templates OR relax the validator to accept a leading SPDX comment before K9!
mirror.yml 56 mixed Bug A (timeout) + stale pre-#416 reds + skipped-misread-as-failure strip timeout-minutes; the rest is expected-noise

Remediation plan (highest leverage first)

  1. Strip timeout-minutes: from every reusable-caller (uses:) job (Bug A) + fix the wrapper generator so it stops emitting it. Detector: actionlint. → clears Governance + much of Secret Scanner + Mirror.
  2. Fix Bug B: declare needed write scopes at the reusable top-level permissions: in scorecard-reusable.yml + secret-scanner-reusable.yml (and/or caller workflow-level), then re-pin estate-wide. → clears Scorecard + clean-wrapper Secret Scanner.
  3. || true on the hypatia scan line. → clears ~47.
  4. Drop stale required contexts (e.g. scan / trufflehog, renamed) from each ruleset that still lists them (owner-gated, per-repo) → clears the masked deadlock on the ~207.
  5. K9 template migration (owner-gated) → dogfood-gate.

Items 1–3 are single central edits (reusables + generator) that clear the large majority of the ~300 red repos; 4–5 are owner-gated/per-repo. After 1–3, re-run the audit and reassess the deadlock before touching rulesets.

Activity

  1. hyperpolymath commented on Aug 7, 2026

    @hyperpolymath
    OwnerAuthor

    Re-measured 2026-08-07 — still live, and the remediation has barely moved.

    This issue diagnoses @d135b05 as the master cause of estate CI-red, affecting
    ~300 repos. Checking whether that pin has actually been rolled forward:

    $ gh api "search/code?q=org:hyperpolymath+d135b05bfc647d0c0fbfedc7e80f37ea50f49236" --jq .total_count
    270
    

    270 files across the org still reference that exact SHA. The pin that this
    issue identifies as the master cause is still the pin in force nearly everywhere.

    Two things follow:

    1. This is not stale and should not be closed. The diagnosis holds; the
      remediation is what has not happened.
    2. Nobody has been re-measuring it. The issue has carried "~300 repos" since
      2026-06-25 with no recorded movement in either direction.

    That second point is the reason for #591 — a Debtfile
    entry with a probe and a ceiling, so a number like this gets re-checked weekly
    instead of believed. A suitable entry:

    ### stale-reusable-pin-d135b05
    - description: Files still pinned to the reusable SHA identified as the estate CI-red master cause (#426)
    - probe: git grep -l 'd135b05bfc647d0c0fbfedc7e80f37ea50f49236' -- . | wc -l
    - ceiling: <per-repo count>
    - severity: high
    - policy: remediable
    - tri: eliminate
    - tracking: hyperpolymath/standards#426
    
  2. 128 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions