Scorecard reconciler fails correct repos: gh actions-lock --verify is blind to job-level reusable refs (wrong in both directions)

[hyperpolymath]

Summary

gh actions-lock --verify (v0.1.6) cannot see job-level uses: — reusable-workflow
calls. It sees only step-level refs. It is therefore wrong in both directions on the
same ref class, and scripts/reconcile-scorecard-actions-lock.rb turns the false-positive
direction into a red OSSF Scorecard for repos that have no defect.

Index case: metadatastician/burble. Its Scorecard has been startup_failure (separate
cause, cured in metadatastician/burble#231); the moment it could actually run, it failed at
reconcile with:

Scorecard reconciliation failed: Native action-lock verification failed for
  .github/workflows/release.yml

The tool's claim, and why it is false

stale — lockfile pins slsa-framework/slsa-github-generator@f7dd8c54… but no
uses: in this workflow references it

release.yml:144 references it plainly:

    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@f7dd8c54c2067bafc12ca7a55595d5ee9b75204a # v2.1.0

The lock entry is correct as written. The lock records owner/repo@ref with the action
subpath normalised away — that is the lock format's own spelling and it is what GitHub
accepts. Deleting the entry, as the tool's remediation advises, would break the lock.

Root cause — four-arm control, run locally against the real tree

arm	tree	verdict
A	release.yml as committed (step refs + 1 job-level slsa ref)	valid:false stale
B	trailing # v2.1.0 comment stripped	valid:false — comment refuted
C	slsa dep record given a uses: [] closure	valid:false — missing closure refuted
D	scorecard.yml job-level ref repointed to a SHA absent from the lock	valid:true, findings []

Arm D is the proof and the alarming half. A workflow whose only ref is a job-level
reusable call verifies vacuously — zero refs extracted, so nothing to compare. The tool
passed a workflow pinning a SHA its lock does not record.

So burble's governance.yml, hypatia-scan.yml and scorecard.yml verify green for the
wrong reason, not because the tool handled them.

The exact trigger for the false stale

Mixed workflow (step-level refs the tool does see) AND the per-workflow lock list
contains an entry for the job-level ref (which it does not see) — that entry is then left
unmatched and reported stale.

Census over the 40-repo estate clone set: 8 mixed workflows, of which the tool actually
rejects 2. Not 8 — the other 6 differ on the second condition.

repo	workflow	job-level ref in per-workflow lock list?	tool
metadatastician/burble	release.yml	yes (slsa-framework/…)	valid:false stale
hyperpolymath/ephapax	rust-ci.yml	yes (hyperpolymath/standards@571cc734)	valid:false stale
hyperpolymath/kitchenspeak	release.yml	no	valid:true
hyperpolymath/session-sentinel	release.yml	no	valid:true
hyperpolymath/llm-grace	release.yml	no	valid:true
hyperpolymath/methodologies	release.yml	no	valid:true
hyperpolymath/hybrid-automation-router	release.yml	no	valid:true
hyperpolymath/universal-chat-extractor	secret-scanner.yml	n/a	valid:false unreachable-pin (different defect)

sha-as-ref findings do not set valid:false; only stale does. That is why 6 other
burble workflows carry findings and still pass.

A second, separate question this surfaced — NOT answered here

The five repos marked "no" above each have a release.yml whose lock key exists but
whose list omits the slsa job-level ref the workflow uses. Whether GitHub enforces the
lock for job-level reusable refs, and therefore whether those runs would be rejected at
startup, is untested:

• Four of the five have never run release.yml — it is tag-triggered.
• hybrid-automation-router did run, and did not startup-fail — but that run
  (2bc990a8, 2026-08-27T05:02:50Z) predates its own lock commit 6f900c86
  (05:11:24Z) by nine minutes, and its current lock 816c2bfb by three weeks. It is not
  evidence either way.

So this is a latent risk on 5 repos whose next tag is the first real test. It needs its
own controlled experiment and is not claimed as a defect here.

Impact on this repo

scripts/reconcile-scorecard-actions-lock.rb (line ~58) shells out to:

Open3.capture3('gh', 'actions-lock', relative, '--verify', '--no-interactive',
               '--json=valid,findings', chdir: root)
raise "Native action-lock verification failed for #{relative}" unless status.success? && ...

and raises on valid:false. The reusable then fails by design. So a known-false verdict
from a blind verifier becomes a red security check on a correct repository — and, per arm D,
the same verifier cannot detect the real desync class this estate's lock campaign exists to
fix.

Acceptance criteria

• AC1 — The reconciler does not fail a repo on a stale finding whose dependency
  is referenced by a job-level uses: in that workflow. Either pre-filter such
  findings, or stop treating stale as fatal.
• AC2 — A regression control exists carrying both directions: a mixed workflow that must
  not be reported stale, and an arm-D tree (job-level ref pinned to a SHA absent from
  the lock) that must not be reported valid. (AC2 is the durable half — AC1 alone
  re-breaks as soon as the tool changes shape.)
• AC3 — metadatastician/burble and hyperpolymath/ephapax reach a green Scorecard
  without any change to their actions.lock or workflow YAML, since neither is
  defective.
• AC4 — The upstream defect is filed against gh-actions-lock with the four-arm control.
  (This is item (e) of the seven upstream defects already ruled to be filed separately.)
• AC5 — The job-level-enforcement question above is settled by a controlled experiment,
  or explicitly recorded as open. It must not be closed by AC1–AC4 evidence.

Do not "repair" the accused repos

The accusation is the bug. burble's and ephapax's locks are correct; removing the flagged
entry would introduce the very desync the lock exists to prevent.

🤖 Generated with Claude Code

https://claude.ai/code/session_01X3hgXxWm6umMgZkjYyHnnm

[hyperpolymath]

AC5 answered — and it forces me to withdraw half of this issue's severity claim

I said above that the job-level enforcement question was open. It is now closed, by
a two-arm control measured 2026-09-23. The answer is no: GitHub does not enforce
actions.lock for job-level reusable-workflow refs. One direction of the defect I
reported therefore does not exist, and I am retracting it.

The control

arm	repo / workflow	job-level ref in its per-workflow lock list?	run at that exact HEAD
negative	hyperpolymath/seamstress scorecard.yml @ 6c7e183c	no — list is []	success, 2026-09-23T04:18:55Z
negative (corrob.)	hyperpolymath/maa-framework secret-scanner.yml @ cdafd055	no — list is []	success, 2026-09-22T23:33:52Z
positive	hyperpolymath/awesome-nickel codeql.yml @ 39d49911	step-level mismatch	startup_failure, 2026-09-23T06:25:39Z

seamstress/scorecard.yml is the clean case: its only uses: anywhere in the file is
the job-level reusable ref on line 18

uses: hyperpolymath/standards/.github/workflows/scorecard-reusable.yml@84355587cb2a1f86e6882de83514a32db2646e7a

and its lock records nothing at all for that path. GitHub ran it anyway, successfully,
this morning. The positive arm rules out "enforcement is simply off in this estate": a
step-level literal mismatch was still being rejected at startup on the same day.

What this retracts

I claimed arm D of the four-arm control showed the verifier is "wrong in both directions —
false stale AND false pass on the exact desync GitHub startup-fails." The second half is
withdrawn. A job-level lock↔YAML mismatch is not a desync GitHub rejects, so the verifier
passing it vacuously is a blind spot, not a missed kill. Arm D still proves the extraction
is vacuous — that remains the mechanism — but it has no production consequence of its own.

The defect that survives is the false stale, and that one is unchanged and real: the
Scorecard reusable fails by design on correct repositories, which is what AC1–AC4 address.

What this also retracts, and it is the bigger number

A census of the whole repin corpus finds 59 job-level refs across 16 repos whose
per-workflow lock list does not record them — not the 5 I previously described. Under the
measurement above, none of these is a defect and none needs repairing. This makes the
"Do not repair the accused repos" warning in this issue considerably more load-bearing than
when I wrote it: acting on the tool's remediation across that population would have
introduced real desyncs into 16 repositories that are currently correct.

Two smaller corrections

• I wrote that only stale sets valid:false. My own census table contradicts that in the
  same issue: universal-chat-extractor is valid:false on unreachable-pin. Correct
  statement: sha-as-ref does not set valid:false; stale and unreachable-pin do.
• hyperpolymath/ephapax is confirmed non-defective, now on the SHA rather than by
  assumption — rust-ci.yml pins standards@571cc734cd69fb846032ec77a662aa8ee4fc32cd and the
  lock records exactly that ref. My earlier census compared owner/repo only, having split the
  SHA off, so it could not have told the difference. The stale verdict there is a pure false
  positive, as AC3 states.

AC5 is therefore closed as answered

AC5: determine whether GitHub enforces actions.lock for job-level reusable refs.

Answered: it does not. Enforcement is step-level only. AC1–AC4 stand.

Unrelated and filed separately if it reproduces: the positive-control arm shows
awesome-nickel/codeql.yml is startup_failure again on 39d49911 as of this morning,
having been cured during the lock campaign.

🤖 Generated with Claude Code

https://claude.ai/code/session_01X3hgXxWm6umMgZkjYyHnnm