Skip to content

actions.lock step-level desync: 39 repos with silently dead CI (startup_failure) #968

Description

@hyperpolymath

Summary

An estate-wide census of all 200 repos carrying .github/workflows/actions.lock finds
39 repos whose CI is silently dead — GitHub rejects their runs at startup (jobs=0,
conclusion startup_failure), with no red X on any job because no job is ever created.

Root cause is the one proven in metadatastician/burble#223: a workflow contains a step-level
uses: ref that is not recorded in actions.lock under that workflow's own path. Dependabot
rewrites refs in workflow YAML and never touches the lockfile, so every grouped github-actions
update desynchronises the repo.

The headline number is a trap — read the split

198 of 200 repos are technically desynced. That figure is close to useless. The split is:

class repos consequence
STEP-level ref missing 39 startup death — CI silently dead
JOB-level reusable ref only (hyperpolymath/standards@…) 159 lock incomplete, CI runs normally
in sync 1 (burble, post-#224) —
transfer redirect (hyperpolymath/007 → 007-lang) 1 —

A missing job-level reusable ref does not kill a run, and this is measured, not assumed:

  • hyperpolymath/standards mirror.yml has a literally empty lock entry
    ('.github/workflows/mirror.yml': []) while calling
    hyperpolymath/standards/.github/workflows/mirror-reusable.yml@571cc734 — its latest run
    created 7 jobs.
  • A random sample of 12 of the 159 cross-repo job-only repos: 12/12 created jobs
    (11 failure, 1 success), zero startup_failure.

So the 159 are a lockfile-completeness gap, tracked separately — not a live outage.

Evidence for the 39

For each repo, the latest run of a workflow carrying a step-level miss:

conclusion repos
startup_failure 37
stale run predating the desync 2

The two exceptions are hyperpolymath/ephapax (push-email-notify.yml, last ran 2026-07-17,
skipped) and hyperpolymath/universal-chat-extractor (codeql.yml, last ran 2026-07-30,
failure). Both are still desynced and will die on their next run. There are no live
counterexamples.

⚠ Method note: an earlier pass sampled the first flagged workflow per repo and produced two
apparent counterexamples (hyperpolymath/servers, metadatastician/first-post). Both dissolved —
their first flagged file is JOB-only; the step-level miss is in a different workflow. Sample a
workflow that carries the defect you are testing for.

Affected repos

repo step-misses sampled workflow latest run
hyperpolymath/hermeneia 54 codeql.yml 2026-09-21 startup_failure
hyperpolymath/servers 26 python.yml 2026-09-20 startup_failure
metadatastician/common-signal 11 codeql.yml 2026-09-21 startup_failure
hyperpolymath/marid 9 capability-spec.yml 2026-09-22 startup_failure
metadatastician/paint-type 8 codeql.yml 2026-09-18 startup_failure
metadatastician/sim-public-relations 7 codeql.yml 2026-09-17 startup_failure
hyperpolymath/my-lang 7 cflite_batch.yml 2026-09-20 startup_failure
metadatastician/svalinn 6 codeql.yml 2026-09-19 startup_failure
metadatastician/harvard-dehallucinator 6 codeql.yml 2026-09-22 startup_failure
metadatastician/pong-ping 5 codeql.yml 2026-09-12 startup_failure
metadatastician/chronicles-of-slavia 5 codeql.yml 2026-09-21 startup_failure
hyperpolymath/verisimdb 5 codeql.yml 2026-09-22 startup_failure
hyperpolymath/this-ssg 5 codeql.yml 2026-09-12 startup_failure
metadatastician/gossamer 4 android-ndk.yml 2026-09-15 startup_failure
metadatastician/cadastra 4 codeql.yml 2026-09-17 startup_failure
hyperpolymath/rpa-elysium 4 cflite_batch.yml 2026-09-20 startup_failure
hyperpolymath/ephapax 4 push-email-notify.yml 2026-07-17 skipped
hyperpolymath/boj-server-cartridges 4 build.yml 2026-09-22 startup_failure
metadatastician/universal-modding-studio 3 codeql.yml 2026-09-20 startup_failure
metadatastician/idaptik-ums 3 codeql.yml 2026-09-20 startup_failure
metadatastician/enaction-engine 3 codeql.yml 2026-09-17 startup_failure
hyperpolymath/vexometer 3 codeql.yml 2026-09-21 startup_failure
hyperpolymath/rsr-template-repo 3 codeql.yml 2026-09-15 startup_failure
hyperpolymath/krl 3 fragment-conformance.yml 2026-09-21 startup_failure
hyperpolymath/blocky-writer 3 codeql.yml 2026-09-22 startup_failure
hyperpolymath/action-trust-layers 3 codeql.yml 2026-09-21 startup_failure
metadatastician/first-post 2 pages.yml 2026-09-15 startup_failure
hyperpolymath/universal-chat-extractor 2 codeql.yml 2026-07-30 failure
hyperpolymath/social-media-tools 2 dogfood-gate.yml 2026-09-21 startup_failure
hyperpolymath/Skein.jl 2 codeql.yml 2026-09-22 startup_failure
hyperpolymath/session-sentinel 2 codeql.yml 2026-09-21 startup_failure
hyperpolymath/laniakea 2 codeql.yml 2026-09-22 startup_failure
hyperpolymath/kitchenspeak 2 codeql.yml 2026-09-22 startup_failure
hyperpolymath/julia-professional-registry 2 codeql.yml 2026-09-22 startup_failure
hyperpolymath/gitbot-fleet 2 codeql.yml 2026-09-22 startup_failure
hyperpolymath/bofj-kitt 2 codeql.yml 2026-09-22 startup_failure
hyperpolymath/awesome-nickel 2 codeql.yml 2026-09-22 startup_failure
hyperpolymath/road-skate 1 dogfood-gate.yml 2026-09-21 startup_failure
hyperpolymath/cicd-squabbler 1 dependabot-automerge.yml 2026-09-21 startup_failure

Why regeneration alone is not the fix

37 of the 39 take grouped github-actions Dependabot updates (gha=yes, grouped=yes).
Dependabot cannot update actions.lock, so a one-off regeneration buys one week; each repo
re-breaks on its next merged Dependabot PR.

The recurrence cure already shipped in burble (#224): a lock-sync gate workflow carrying
zero uses: — checkout via git clone in a run: step rather than actions/checkout — so
the detector is structurally immune to the very desync it detects, and it fails the Dependabot PR
before merge. It must fail the job: not continue-on-error, not a ::warning::, which cannot
fail a job.

Acceptance criteria — per repo

For each of the 39, the repo is done when all of the following hold:

  1. gh actions-lock --no-migrate-local-actions --no-narrow regenerated, and
    gh actions-lock --verify-local is clean offline.
  2. grep -c '\$/' .github/workflows/*.yml → 0 (no local-composite-action rewrite;
    --no-migrate-local-actions is mandatory — fix mode has corrupted locks before).
  3. git diff of actions.lock shows every pre-existing SHA surviving, or an intentional bump.
    No tag-for-SHA de-pinning.
  4. Every previously step-flagged workflow reports jobs > 0 on the PR head.
  5. The zero-uses: lock-sync gate is present and fails the job on desync.
  6. Any workflow that goes red for a real reason once it starts running again becomes its own
    issue — it does not block the lock fix (estate stopping rule).

Scope

This issue covers the 39 step-level repos only. The 159 job-level-only repos are tracked
separately and require no repo changes.

Census artefact: 200 rows × 13 columns, sampled on each repo's default branch via one GraphQL
call per repo (never a local working copy — which checkout you sample is the measurement).

🤖 Generated with Claude Code

https://claude.ai/code/session_01X3hgXxWm6umMgZkjYyHnnm

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions