From ba0947b4dba6f4c240d1b15e4cc2abe8303bfcfe Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Sun, 20 Sep 2026 22:46:20 +0000 Subject: [PATCH] =?UTF-8?q?fix(audit,gate):=20withdraw=20the=20five=20resi?= =?UTF-8?q?dues=20=E2=80=94=20a=20file-based=20census=20cannot=20see=20Git?= =?UTF-8?q?Hub-generated=20checks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The five contexts this census flagged as unsatisfiable are all published: every one reports in the check-run history of the repository that requires it, as a job of a GitHub-generated workflow (code scanning default setup and friends) that no file in the repository mentions. Evidence, gathered 2026-09-20 from check-runs on the default branch: awesome-gleam Analyze (actions) 8 of the last 8 commits, and PR #12's head awesome-gleam Adjust Configuration reports on main (intermittent) casket-ssg Analyze Code (actions) 15 of the last 30 commits coord-tui CodeQL Analysis (actions, none) 5 of the last 8 commits, and PR #63's head vext CodeQL Analysis (actions, none) 12 reports over 8 commits, and PR #51's head The previous commit stated that all five were re-checked and are "live defects". That re-check answered the wrong question: it confirmed the *rules* were live (active enforcement, ~DEFAULT_BRANCH, integration_id 15368) and never asked whether a *publisher* exists. The publisher is the platform. This matters beyond the ledger: merged guidance to go and patch four repositories' required-status-check lists would have deleted working gates on false evidence, and the deleted names would have been the ones nothing could see — the checks GitHub generates itself. Changes: * the audit's results now read 0 unsatisfiable, with the five candidates and their evidence; the residues table becomes the refuted-candidates table; * the method note becomes the operational rule: a file-based pass may only produce candidates; an unsatisfiable verdict becomes a finding only after check-run history confirms the name never reports; a required context is never deleted on a file-based verdict alone; * the gate carries the same limit in its header, in its warning and error text, and in --print-commands, so the person reading a finding is told to confirm it before acting; * the census caveats section cites this as the worked example rather than a hypothetical. The class remains real — tropical-types#17 was a genuine mis-named requirement. What changes is the confidence level a file-based census may claim. --- ...t-ci-context-producibility-2026-09-20.adoc | 59 +++++++++++++------ scripts/check-required-contexts.sh | 17 +++++- 2 files changed, 55 insertions(+), 21 deletions(-) diff --git a/docs/audits/audit-ci-context-producibility-2026-09-20.adoc b/docs/audits/audit-ci-context-producibility-2026-09-20.adoc index e90252c7c..1bbee4db7 100644 --- a/docs/audits/audit-ci-context-producibility-2026-09-20.adoc +++ b/docs/audits/audit-ci-context-producibility-2026-09-20.adoc @@ -86,16 +86,38 @@ status checks through legacy branch protection): | 4 | GitHub-app-published names that carry no integration id (`github-advanced-security`, `Dependabot`) -| *unsatisfiable* -| *5* -| no workflow here publishes it and no app integration is bound to it +| unsatisfiable +| *0* +| see below — the five candidates this pass flagged were refuted by check-run history |=== -That is a good result for the estate and a poor one for the five rules: the class -is rare, which is exactly why it hides. All five carry `integration_id: 15368` -(GitHub Actions), so they are repository-owned *workflow job names* rather than -third-party app names — nothing outside the repository could ever publish them. -The residues, all in auxiliary (`Optimus-Branch`, `Base`) rulesets: +The file-based pass flagged five required contexts as candidates. **All five were +refuted** when put to the check-run history on 2026-09-20: each one reports on the +repository in question, published by a *GitHub-generated* workflow (code scanning +default setup and friends) that no file in the repository mentions: + +[cols="1,1,1,1"] +|=== +| Repository | Candidate context | Evidence in check-run history | Publisher + +| `awesome-gleam` | `Analyze (actions)` | reports on 8 of the last 8 commits, and on PR #12's head | GitHub code scanning (dynamic) +| `awesome-gleam` | `Adjust Configuration` | reports on `main` (intermittent) | GitHub code scanning (dynamic) +| `casket-ssg` | `Analyze Code (actions)` | reports on 15 of the last 30 commits | GitHub code scanning (dynamic) +| `coord-tui` | `CodeQL Analysis (actions, none)` | reports on 5 of the last 8 commits, and on PR #63's head | GitHub code scanning (dynamic) +| `vext` | `CodeQL Analysis (actions, none)` | reports 12 times across the last 8 commits, and on PR #51's head | GitHub code scanning (dynamic) +|=== + +That is the finding this audit exists to publish: **a census over repository files +cannot see the checks GitHub generates itself.** The rule that follows is +operational, not cosmetic — + +* a file-based pass may only produce *candidates*; +* an `unsatisfiable` verdict becomes a finding only after the check-run history of + the default branch confirms the name never reports; +* and a required context is never deleted on a file-based verdict alone. + +The five entries previously listed here as live defects, in four auxiliary +(`Optimus-Branch`, `Base`) rulesets, were withdrawn for exactly that reason: [cols="2,1,2,2"] |=== @@ -108,17 +130,16 @@ The residues, all in auxiliary (`Optimus-Branch`, `Base`) rulesets: | `hyperpolymath/vext` | Optimus-Branch | `CodeQL Analysis (actions, none)` | stale job-name + matrix tuple |=== -Each one was re-checked per ruleset on 2026-09-20 and is a live defect, not an -artefact of evaluating the wrong ref: every carrying ruleset is `active`, every -one binds `~DEFAULT_BRANCH` (these repositories have a single branch, -`main`), and every context carries `integration_id: 15368`. Note the naming -irony in three of them — a ruleset called `Optimus-Branch` guarding the default -branch. - -Each one is the same defect as tropical-types#17 in a different costume: the -requirement names a check that used to exist. The fix is one of the two honest -options below; the diagnostic is -`scripts/check-required-contexts.sh --print-commands`. +The per-ruleset check was correct as far as it went — every carrying ruleset is +`active`, every one binds `~DEFAULT_BRANCH`, and every context carries +`integration_id: 15368` — but it answered the wrong question: whether the *rule* +was live, not whether a *publisher* exists. The publisher is a GitHub-generated +workflow, invisible to the repository and to this census. + +The revised method, and the cost of getting it wrong, are the same lesson as +tropical-types#17: a name that nothing in the repository publishes can still be +satisfied by the platform, and a name that nothing publishes *anywhere* is a +blocked pull request nobody can diagnose from the CI board. == Case study — tropical-types#17 diff --git a/scripts/check-required-contexts.sh b/scripts/check-required-contexts.sh index cfd0cb1cf..7110d6dad 100755 --- a/scripts/check-required-contexts.sh +++ b/scripts/check-required-contexts.sh @@ -18,6 +18,15 @@ # bare one), and catalogued for the wrapper case in # docs/audits/audit-hypatia-pin-orphan-2026-05-27.adoc. # +# LIMIT — READ THIS BEFORE ACTING ON A VERDICT +# -------------------------------------------- +# A verdict here is derived from the repository's files. GitHub also generates +# workflows server-side (code scanning default setup and friends) and those +# publish check names nothing in the repository mentions. A name this script +# calls unsatisfiable may therefore be reporting on every commit. Confirm any +# such verdict against the check-run history of the default branch, and never +# delete a required context on this script's word alone. +# # THE RULE THIS ENCODES # --------------------- # * a plain job publishes its `name:` — or ` ()` when it @@ -239,9 +248,9 @@ run_audit() { UNSATISFIABLE) unsatisfiable=$((unsatisfiable+1)) if [ "$STRICT" = 1 ]; then - err "required context '$ctx' ($src) is not producible by any workflow here, is not bound to an app integration, and no job publishes a matching name" + err "required context '$ctx' ($src) is not producible by any workflow here, is not bound to an app integration, and no job publishes a matching name (candidate only: GitHub-generated workflows publish checks no file mentions — confirm against the check-run history before changing the rule)" else - warn "required context '$ctx' ($src) is not producible by any workflow here and is not bound to an app integration — unsatisfiable as written" + warn "required context '$ctx' ($src) is not producible by any workflow here and is not bound to an app integration — unsatisfiable as written (candidate only: GitHub-generated workflows publish checks no file mentions — confirm against the check-run history before changing the rule)" fi if [ "$PRINT_COMMANDS" = 1 ]; then cat < / '). + # FIRST, though: confirm the name never reports. Read the check-run history of + # the default branch — GitHub-generated workflows publish checks that no file + # in this repository mentions, and five such candidates were refuted this way + # on 2026-09-20 (docs/audits/audit-ci-context-producibility-2026-09-20.adoc). EOS fi ;; app-owned) info "app-owned ($src): $ctx" ;;