From f2146fa3718843893a864a91f940f3935b97ca8b Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Sun, 20 Sep 2026 21:53:11 +0000 Subject: [PATCH] feat(ci): gate on required status contexts nothing can publish MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GitHub matches a required status check by string equality against the name it publishes for that run, and never checks that the name is producible. A rule requiring a name no publisher can supply is unsatisfiable: every pull request stays BLOCKED while the CI board shows green, and nothing on the CI side names the cause. That is the general form of hyperpolymath/tropical-types#17 (the bare `Hypatia Neurosymbolic Analysis` required while the wrapper published `scan / Hypatia Neurosymbolic Analysis`) and of the wrapper-prefix mismatch recorded in docs/audits/audit-hypatia-pin-orphan-2026-05-27.adoc. Adds, all read-only and dependency-free (bash + awk + jq + gh + curl): * scripts/check-required-contexts.sh — derives the names a repository's workflows can publish (job display names, ` (*)` matrix forms, and ` / ` for reusable-wrapper callers), reads the required contexts from every enabled branch ruleset (and legacy branch protection when the credential can read it), and reports any required context that neither a workflow nor a bound app integration can satisfy. App-bound contexts are skipped by design: CodeRabbit, SonarCloud and the code-scanning app own their names. Advisory by default (REQUIRED_CONTEXTS_STRICT=1 to gate), and an API failure is reported as "not inspected" rather than "nothing is required" — a gate that fails open on the failure of the call it depends on is worse than no gate. * scripts/tests/check-required-contexts-test.sh — drives the built-in hermetic fixture: a synthetic repository, a stubbed reusable in a cache directory, and JSON required-context sets. No network, no token, no API. Five branches, including a regression case for exact-versus-prefix matching. * docs/audits/audit-ci-context-producibility-2026-09-20.adoc — the audit that produced the gate. Census of all 365 non-archived public repositories: 337 required contexts, of which 168 are app-owned, 156 producible, 4 app-named and 5 unsatisfiable across 4 repositories (awesome-gleam, casket-ssg, coord-tui, vext — every one a retired CodeQL-era job name). Records the tropical-types case study, the two honest remediations for a bare-name requirement (with the defiant inline-implementation precedent), the caller-id drift across the estate (200 `scan` versus 8 `hypatia`), the caveats (legacy branch protection needs Administration: read; matrix forms are matched as a family), and the adjacent RE001-versus-allowlist inconsistency (step-security/harden-runner is not in the allowed-actions list the same estate enforces). Deliberately not wired into governance-reusable.yml: that changes the check surface of every repository pinned to it, and the audit records the shape and the advisory-first rationale instead. --- ...t-ci-context-producibility-2026-09-20.adoc | 239 +++++++++++++ scripts/check-required-contexts.sh | 327 ++++++++++++++++++ scripts/tests/check-required-contexts-test.sh | 40 +++ 3 files changed, 606 insertions(+) create mode 100644 docs/audits/audit-ci-context-producibility-2026-09-20.adoc create mode 100755 scripts/check-required-contexts.sh create mode 100755 scripts/tests/check-required-contexts-test.sh diff --git a/docs/audits/audit-ci-context-producibility-2026-09-20.adoc b/docs/audits/audit-ci-context-producibility-2026-09-20.adoc new file mode 100644 index 000000000..5def65dbe --- /dev/null +++ b/docs/audits/audit-ci-context-producibility-2026-09-20.adoc @@ -0,0 +1,239 @@ +// SPDX-License-Identifier: CC-BY-SA-4.0 +// SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) += Audit — required status contexts that nothing can publish +:toc: +:toclevels: 2 + +[abstract] +GitHub matches a required status check by *string equality* against the name it +publishes for that check run. It never checks that the name is *producible*. A +rule requiring a name no publisher can supply is an unsatisfiable requirement: +every pull request stays `BLOCKED`, the CI board shows green, and nothing on the +CI side names the cause. This audit defines the class, measures it across the +estate, records the residues, and ships the gate that keeps it from returning +(`scripts/check-required-contexts.sh`). + +Sibling audits: `audit-hypatia-pin-orphan-2026-05-27.adoc` (the orphan-SHA class +and the wrapper-prefix mismatch this audit generalises), +`audit-hypatia-cache-sha-corruption-2026-06-21.adoc` (the false-green cache). + +== The rule + +[cols="1,2"] +|=== +| Publisher | Name published + +| a plain job +| its `name:` — or ` ()` when it has a matrix and no `name:` — or the job id alone + +| a job that calls a reusable workflow +| ` / ` + +| a third-party app (code scanning, CodeRabbit, SonarCloud, …) +| whatever the app publishes; the repository does not control it +|=== + +Two consequences carry the whole class: + +. **A wrapper can never publish the reusable's bare inner name.** GitHub always + prefixes it with the caller. Requiring the bare name from a wrapped gate is + unsatisfiable by construction — this is issue #17 in + `hyperpolymath/tropical-types`. +. **A required name survives the workflow it named.** Rename a job, retire a + workflow, or change how a reusable names its jobs, and the requirement silently + becomes unsatisfiable while every check that *does* run stays green. + +== Method + +For every non-archived public repository in `hyperpolymath` (365 on 2026-09-20), +and for each repository with at least one required status context: + +. derive the set of names its workflows can publish — job display names, matrix + forms (` (*)`), and, for reusable callers, ` / ` + resolved from the pinned ref; +. read the required contexts from every enabled branch ruleset, keeping the + `integration_id` GitHub records for each; +. classify each required context: + * *app-owned* — bound to an integration other than GitHub Actions; that app + owns the name; + * *producible* — some workflow here publishes exactly that name (or a + wildcard form of it); + * *unsatisfiable* — neither, so no run will ever satisfy it. + +App-bound contexts are skipped by design: requiring `CodeRabbit` or +`SonarCloud Code Analysis` is legitimate, and the repository has no control over +those names. The audit's subject is the residue the *repository itself* owns. + +== Results + +337 required contexts across 179 repositories, read from branch rulesets: + +[cols="1,1,2"] +|=== +| Verdict | Count | Meaning + +| app-owned +| 168 +| published by a non-Actions integration; not the repository's to supply + +| producible +| 156 +| a workflow in the repository publishes exactly this name + +| app-named +| 4 +| GitHub-app-published names that carry no integration id (`github-advanced-security`, `Dependabot`) + +| *unsatisfiable* +| *5* +| no workflow here publishes it and no app integration is bound to it +|=== + +That is a good result for the estate and a poor one for the five rules: the class +is rare, which is exactly why it hides. The residues, all in auxiliary +(`Optimus-Branch`, `Base`) rulesets: + +[cols="2,1,2,2"] +|=== +| Repository | Ruleset | Required context | Reading + +| `hyperpolymath/awesome-gleam` | Base | `Adjust Configuration` | no workflow publishes it; looks like a retired job name +| `hyperpolymath/awesome-gleam` | Base | `Analyze (actions)` | the CodeQL job here does not publish this form +| `hyperpolymath/casket-ssg` | Optimus-Branch | `Analyze Code (actions)` | the CodeQL job was renamed; the rule was not +| `hyperpolymath/coord-tui` | Optimus-Branch | `CodeQL Analysis (actions, none)` | stale job-name + matrix tuple +| `hyperpolymath/vext` | Optimus-Branch | `CodeQL Analysis (actions, none)` | stale job-name + matrix tuple +|=== + +Each one is the same defect as tropical-types#17 in a different costume: the +requirement names a check that used to exist. The fix is one of the two honest +options below; the diagnostic is +`scripts/check-required-contexts.sh --print-commands`. + +== Case study — tropical-types#17 + +* `main`'s rules required the bare `Hypatia Neurosymbolic Analysis`. +* The repository's wrapper published `scan / Hypatia Neurosymbolic Analysis` + (caller job `scan`), and the run-level aggregate `Hypatia` — neither matched. +* Verified 2026-09-20: legacy branch protection on `main` reported + `required_status_checks.enforcement_level = "off"` with an empty `contexts` + list, and the `Base` ruleset carried no `required_status_checks` rule, so + nothing was blocked *and nothing was enforced*. +* Fixed in `hyperpolymath/tropical-types` PR #52 (caller job `scan` → `hypatia`, + so the canonical string is published; plus a repository-local contract and + guard) and PR #53 (the guard job used the secrets context, which Hypatia's + RE001 research extension flags; `step-security/harden-runner` is *not* in the + actions allowlist, so the job now uses `github.token` and reports what it could + not read instead). + +The two names the fix made true are visible in the check-run list on that +repository's main: `hypatia / Hypatia Neurosymbolic Analysis` (wrapper) and +`Required contexts are producible` (the guard). + +== The correct way to satisfy a bare name + +`hyperpolymath/defiant` requires the bare `Hypatia Neurosymbolic Analysis` +(binding `integration_id: 15368` = GitHub Actions) and *satisfies* it — because +its `.github/workflows/hypatia-scan.yml` is an **inline implementation**, and the +job itself carries `name: Hypatia Neurosymbolic Analysis`. Nothing is prefixed +when there is no reusable call. + +So the choice is structural, not cosmetic: + +. require a *prefixed* name — and let the shared reusable publish it + (`hypatia / Hypatia Neurosymbolic Analysis` after naming the caller job + `hypatia`); or +. require a *bare* name — and implement the check in the repository, because a + wrapper cannot produce it. + +== Naming convention drift (worth a decision) + +Across the estate's 208 Hypatia wrappers the caller job id is inconsistent: +`scan` in 200 repositories, `hypatia` in 8. The published context therefore +differs per repository (`scan / …` vs `hypatia / …`), and `audit-hypatia-pin-orphan-2026-05-27.adoc` +already documents `hypatia / Hypatia Neurosymbolic Analysis` as the canonical +string. Two consequences: the canonical string in that audit is wrong for 200 +repositories, and any bulk re-arm of the context would be wrong for one group or +the other. One caller id, recorded per repository, removes the ambiguity — the +gate below reports what each repository actually publishes. + +== The gate + +`scripts/check-required-contexts.sh` — read-only, `gh` + `jq` + `awk` + `curl` +only, no new dependency: + +[source,sh] +---- +bash scripts/check-required-contexts.sh [REPO_ROOT] # advisory +REQUIRED_CONTEXTS_STRICT=1 bash scripts/check-required-contexts.sh . # gating +bash scripts/check-required-contexts.sh . --print-commands # remediation text +bash scripts/check-required-contexts.sh . --json out.json # machine-readable +REUSABLE_CACHE_DIR=… bash scripts/check-required-contexts.sh . # offline cache +---- + +* exit `0` when every required context is satisfiable (or nothing is required); +* exit `1` on an unsatisfiable context **only** under + `REQUIRED_CONTEXTS_STRICT=1` — advisory by default, per the estate rule that a + gate which reds nobody yet is a warning, not a red; +* an API failure (rate limit, missing scope) is reported as *not inspected*, and + never as "nothing is required": a gate that fails open on the failure of the + call it depends on is worse than no gate; +* `scripts/tests/check-required-contexts-test.sh` drives the built-in hermetic + fixture (synthetic repository, stubbed reusable, JSON required sets) — no + network, no token, no API, five branches including a regression case for + prefix-versus-exact matching. + +=== Recommended wiring (not done here, deliberately) + +The check is estate-shaped, so it belongs in `governance-reusable.yml` — but +adding a job there changes the check surface of every repository pinned to it, +and this audit has no mandate for that. The shape it should take when the owner +arms it: + +[source,yaml] +---- + required-contexts: + name: Required contexts are producible + runs-on: ${{ inputs.runs-on }} + continue-on-error: true # advisory first; a fleet-wide red is not a gate + steps: + - uses: actions/checkout@ + - env: + GH_TOKEN: ${{ github.token }} # rulesets are readable without admin + run: bash .standards-checkout/scripts/check-required-contexts.sh . +---- + +== Caveats + +* *Legacy branch protection is not covered by this census.* Reading it needs a + credential with `Administration: read`, which the audit credential did not + have; the script reports that honestly and inspects rulesets only. Re-run it + with such a credential before declaring the class discharged. +* *App-bound contexts are skipped* (`integration_id` present and not GitHub + Actions). A mis-configured app binding is therefore out of scope here. +* *Disabled rulesets are skipped*; a rule that is off cannot block anyone. +* *Matrix forms are matched as ` (*)`* — the audit cannot know the matrix + values without running the workflow, so a matrix job counts as producing the + family. Where a required name names one tuple (`CodeQL Analysis (actions, + none)`), the job itself is what must match. + +== Adjacent finding — RE001 versus the allowlist + +Hypatia's `research_extensions: RE001` asks jobs that reference `secrets.*` to +install `step-security/harden-runner`. That action is absent from +`rhodium-standard-repositories/actions-allowlist/allowed-actions.json` (89 +patterns, no `step-security/*`), so a repository subject to the allowlist cannot +satisfy RE001 by the route the rule suggests without an allowlist change. Either +the rule or the allowlist should move; meanwhile the workable answer is to not +reference the secrets context (use `github.token`, as tropical-types does) or to +route the credentialed work through a reusable, where the caller passes +`secrets: inherit` and no `secrets.*` appears in the job. + +== Provenance + +* Class origin: `hyperpolymath/tropical-types#17`; remediation PRs #52 and #53. +* Sibling audits: `audit-hypatia-pin-orphan-2026-05-27.adoc` (wrapper-prefix + mismatch), `audit-hypatia-cache-sha-corruption-2026-06-21.adoc` (false green). +* Gate: `scripts/check-required-contexts.sh`, fixture + `scripts/tests/check-required-contexts-test.sh`. +* Tooling used for the census: GitHub REST API (`/rulesets`, branch protection), + raw workflow and reusable sources at their pinned refs. diff --git a/scripts/check-required-contexts.sh b/scripts/check-required-contexts.sh new file mode 100755 index 000000000..420c1abf2 --- /dev/null +++ b/scripts/check-required-contexts.sh @@ -0,0 +1,327 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# +# check-required-contexts.sh — is every required status context a name something +# can actually publish? +# +# WHY +# --- +# GitHub matches a required status check by *string equality* against the name it +# publishes for that check run. It never checks that the name is producible. So a +# rule requiring a name nothing publishes is an unsatisfiable requirement: every +# pull request stays BLOCKED, the CI board shows green, and nothing on the CI +# side names the cause. Observed in hyperpolymath/tropical-types#17 (the bare +# `Hypatia Neurosymbolic Analysis` required while the wrapper publishes +# `scan / Hypatia Neurosymbolic Analysis`), and catalogued for the wrapper case in +# docs/audits/audit-hypatia-pin-orphan-2026-05-27.adoc. +# +# THE RULE THIS ENCODES +# --------------------- +# * a plain job publishes its `name:` — or ` ()` when it +# has a matrix and no `name:`, or the job id alone otherwise; +# * a job that calls a reusable workflow publishes +# / +# and therefore can NEVER publish the reusable's bare inner name; +# * a context bound to a third-party app integration (CodeQL scanning, +# CodeRabbit, SonarCloud, …) is that app's to publish, not the repository's. +# +# USAGE +# scripts/check-required-contexts.sh [REPO_ROOT] [--json FILE] [--print-commands] +# [--no-network] [--self-test] +# +# ENV +# GITHUB_REPOSITORY owner/repo (default: `gh repo view`) +# GH_TOKEN / GITHUB_TOKEN read-only credential; rulesets need no admin +# REQUIRED_CONTEXTS_JSON hermetic input: JSON array of +# {"context": "...", "app_id": 0, "source": "..."} +# (used by the fixture test, and for offline audits) +# REUSABLE_CACHE_DIR where fetched reusable files are cached; also the +# test seam that makes reusable resolution hermetic +# REQUIRED_CONTEXTS_STRICT 1 = exit 1 on an unsatisfiable required context. +# Default 0 = ::warning, per the estate rule that a +# check which reds nobody yet is a warning, not a red. +# APP_NAMED_CONTEXTS space-separated extra context names published by +# GitHub apps rather than by a workflow +# +# EXIT 0 = every required context is satisfiable (or nothing is required); +# 1 = at least one required context cannot be published (strict mode), or +# the repository could not be inspected at all. +set -uo pipefail + +ROOT="." +JSON_OUT="" +PRINT_COMMANDS=0 +OFFLINE=0 +SELF_TEST=0 +# A while-loop with real shifting: `shift` inside `for a in "$@"` cannot consume +# the *next* argument, which silently mis-binds an option value to the following +# flag. (Caught by the smoke run, not by the happy-path test.) +while [ $# -gt 0 ]; do + case "$1" in + --json) JSON_OUT="${2:-}"; shift 2 ;; + --print-commands) PRINT_COMMANDS=1; shift ;; + --no-network) OFFLINE=1; shift ;; + --self-test) SELF_TEST=1; shift ;; + -h|--help) sed -n '2,50p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + -*) printf 'unknown option: %s\n' "$1" >&2; exit 2 ;; + *) ROOT="$1"; shift ;; + esac +done + +STRICT="${REQUIRED_CONTEXTS_STRICT:-0}" +CACHE_DIR="${REUSABLE_CACHE_DIR:-}" +APP_NAMED_DEFAULT="github-advanced-security Dependabot Dependabot Updates Code scanning results" +# shellcheck disable=SC2086 # deliberate word splitting: a space-separated list +APP_NAMED="${APP_NAMED_CONTEXTS:-$APP_NAMED_DEFAULT}" +GH_ACTIONS_APP=15368 +RAW="https://raw.githubusercontent.com" + +say() { printf '%s\n' "$*"; } +info() { printf ' %s\n' "$*"; } +note() { printf '::notice::%s\n' "$*"; } +warn() { printf '::warning::%s\n' "$*"; } +err() { printf '::error::%s\n' "$*"; } + +# --------------------------------------------------------------- derivation --- +# Emit "jobIdnameOrDashusesOrDashisMatrix" for one workflow file. +# Empty fields are written as "-": `read` with IFS= collapses runs of IFS +# whitespace, so an empty field would silently shift every later field. +jobs_of() { + awk ' + /^jobs:[[:space:]]*$/ { injobs=1; next } + /^[^[:space:]#]/ { injobs=0 } + injobs && /^ [A-Za-z0-9_.-]+:[[:space:]]*$/ { + if (job != "") printf "%s\t%s\t%s\t%s\n", job, (name==""?"-":name), (uses==""?"-":uses), (mat?"yes":"no") + job=$1; sub(/:$/,"",job); name=""; uses=""; mat=0; next + } + injobs && /^ name:[[:space:]]*/ { + v=$0; sub(/^[[:space:]]*name:[[:space:]]*/,"",v); gsub(/^"|"[[:space:]]*$/,"",v); name=v + } + injobs && /^ uses:[[:space:]]*/ { + v=$0; sub(/^[[:space:]]*uses:[[:space:]]*/,"",v); sub(/[[:space:]]*#.*$/,"",v); uses=v + } + injobs && /^[[:space:]]+matrix:[[:space:]]*$/ { mat=1 } + END { if (job != "") printf "%s\t%s\t%s\t%s\n", job, (name==""?"-":name), (uses==""?"-":uses), (mat?"yes":"no") } + ' "$1" +} + +fetch_reusable() { # owner/repo/workflow@ref -> path on stdout, or empty + local spec="$1" path orr file ref url cache + path="${spec%@*}"; ref="${spec##*@}" + orr="${path%%/.github/workflows/*}" + file=".github/workflows/${path##*/}" + url="$RAW/$orr/$ref/$file" + if [ -n "$CACHE_DIR" ]; then + cache="$CACHE_DIR/$(printf '%s' "$spec" | tr -c 'A-Za-z0-9.' '_')" + if [ -s "$cache" ]; then printf '%s\n' "$cache"; return 0; fi + else + cache="$(mktemp)"; TMP_FILES="${TMP_FILES:-} $cache" + fi + [ "$OFFLINE" = 1 ] && return 1 + if curl -fsSL "$url" -o "$cache" 2>/dev/null; then + printf '%s\n' "$cache" + else + rm -f "$cache"; return 1 + fi +} + +publishable_names() { # repo root -> one name per line + local f id name uses mat inner iid iname stub + for f in "$ROOT"/.github/workflows/*.yml "$ROOT"/.github/workflows/*.yaml; do + [ -f "$f" ] || continue + while IFS=$'\t' read -r id name uses mat; do + [ -n "$id" ] || continue + [ "$name" = "-" ] && name="" + [ "$uses" = "-" ] && uses="" + case "$uses" in + *".github/workflows/"*"@"*) + if stub="$(fetch_reusable "$uses")" && [ -s "$stub" ]; then + while IFS=$'\t' read -r iid iname _u _m; do + [ -n "$iid" ] || continue + [ "$iname" = "-" ] && iname="" + printf '%s / %s\n' "$id" "${iname:-$iid}" + [ -n "$name" ] && printf '%s / %s\n' "$name" "${iname:-$iid}" + done < <(jobs_of "$stub") + else + printf '%s / *\n' "$id" + [ -n "$name" ] && printf '%s / *\n' "$name" + fi ;; + *) + printf '%s\n' "${name:-$id}" + if [ "$mat" = yes ]; then + printf '%s (*)\n' "$id" + [ -n "$name" ] && printf '%s\n' "$(printf '%s' "$name" | sed 's/\${{ *[^}]* *}}/*/g')" + fi ;; + esac + done < <(jobs_of "$f") + done +} + +# ------------------------------------------------------------ required set ---- +required_json() { # emit {"context":..,"app_id":..,"source":..} objects + if [ -n "${REQUIRED_CONTEXTS_JSON:-}" ]; then + [ -f "$REQUIRED_CONTEXTS_JSON" ] || { err "REQUIRED_CONTEXTS_JSON not found: $REQUIRED_CONTEXTS_JSON"; exit 1; } + jq -c '.[] | {context: .context, app_id: (.app_id // 0), source: (.source // "file")}' "$REQUIRED_CONTEXTS_JSON" \ + || { err "REQUIRED_CONTEXTS_JSON is not a JSON array of {\"context\": ...} objects"; exit 1; } + return 0 + fi + command -v gh >/dev/null 2>&1 || { note "gh not installed — cannot read branch rules"; return 1; } + command -v jq >/dev/null 2>&1 || { note "jq not installed — cannot read branch rules"; return 1; } + [ -n "${GITHUB_REPOSITORY:-}" ] || GITHUB_REPOSITORY="$(gh repo view --json nameWithOwner -q .nameWithOwner 2>/dev/null)" + [ -n "${GITHUB_REPOSITORY:-}" ] || { note "no GITHUB_REPOSITORY and gh cannot resolve the current repo"; return 1; } + + local repo="$GITHUB_REPOSITORY" br id list body + # An API failure returns an object ({"message": ...}), not an array. Without + # this check a rate-limited or unauthorised run would emit no contexts and + # read as "nothing is required" — a false clean bill of health on exactly the + # gate that exists to catch unsatisfiable requirements. + list="$(gh api "repos/$repo/rulesets" 2>/dev/null || true)" + if ! printf '%s' "$list" | jq -e 'type == "array"' >/dev/null 2>&1; then + note "could not read branch rulesets for $repo (API error, rate limit, or missing permission) — nothing inspected" >&2 + return 1 + fi + br="$(gh api "repos/$repo" 2>/dev/null | jq -r 'if type=="object" then (.default_branch // "main") else "main" end' 2>/dev/null || echo main)" + for id in $(printf '%s' "$list" | jq -r '.[]? | select(.target=="branch") | .id'); do + body="$(gh api "repos/$repo/rulesets/$id" 2>/dev/null || true)" + printf '%s' "$body" | jq -c ' + if type == "object" and .enforcement != "disabled" then + .rules[]? | select(.type=="required_status_checks") | .parameters.required_status_checks[]? + | {context: (.context|tostring), app_id: (.integration_id // 0), source: "ruleset"} + else empty end' 2>/dev/null + done + body="$(gh api "repos/$repo/branches/$br/protection/required_status_checks" 2>/dev/null || true)" + if printf '%s' "$body" | jq -e 'type == "object" and has("contexts")' >/dev/null 2>&1; then + note "legacy branch protection readable: enforcement_level=$(printf '%s' "$body" | jq -r '.enforcement_level // "unknown"')" >&2 + printf '%s' "$body" | jq -c '.contexts[]? | {context: ., app_id: 0, source: "branch-protection"}' 2>/dev/null + printf '%s' "$body" | jq -c '.checks[]? | {context: (.context|tostring), app_id: (.app_id // 0), source: "branch-protection"}' 2>/dev/null + else + note "legacy branch protection unreadable with this credential (needs Administration: read) — rulesets only" >&2 + fi + return 0 +} + +# ------------------------------------------------------------------ audit ----- +run_audit() { + local pubb required ROWS unsatisfiable=0 line ctx app verdict + pubb="$(mktemp)"; required="$(mktemp)"; ROWS="$(mktemp)" + publishable_names | sort -u >"$pubb" + required_json >"$required" || true + + say "" + say "== publishable context names ($(wc -l <"$pubb" | tr -d ' '))" + sed 's/^/ /' "$pubb" + say "" + say "== required contexts vs publishable names" + if [ ! -s "$required" ]; then + info "no required status contexts found (or none could be read)" + fi + while IFS= read -r line; do + [ -n "$line" ] || continue + ctx="$(printf '%s' "$line" | jq -r '.context')" + app="$(printf '%s' "$line" | jq -r '.app_id')" + local src; src="$(printf '%s' "$line" | jq -r '.source')" + if [ "$app" != 0 ] && [ "$app" != "$GH_ACTIONS_APP" ]; then + verdict=app-owned + elif awk -v c="$ctx" ' + { if ($0 == c) { ok=1; exit } # exact name + n=$0; gsub(/\*/,".*",n) # matrix/wrapper wildcard form + if (n ~ /\*/ && c ~ ("^" n "$")) ok=1 } + END { exit(ok?0:1) }' "$pubb"; then + verdict=producible + else + verdict=UNSATISFIABLE + fi + printf '%s\t%s\t%s\n' "$ctx" "$app" "$verdict" >>"$ROWS" + case "$verdict" in + UNSATISFIABLE) + unsatisfiable=$((unsatisfiable+1)) + if [ "$STRICT" = 1 ]; then + err "required context '$ctx' ($src) is not producible by any workflow here, is not bound to an app integration, and no job publishes a matching name" + else + warn "required context '$ctx' ($src) is not producible by any workflow here and is not bound to an app integration — unsatisfiable as written" + fi + if [ "$PRINT_COMMANDS" = 1 ]; then + cat < / '). +EOS + fi ;; + app-owned) info "app-owned ($src): $ctx" ;; + *) info "$verdict ($src): $ctx" ;; + esac + done <"$required" + + if [ -n "$JSON_OUT" ]; then + jq -Rs --arg repo "${GITHUB_REPOSITORY:-$ROOT}" --argjson n "$unsatisfiable" --rawfile _ /dev/null ' + {repository: $repo, unsatisfiable: $n, + results: (split("\n") | map(select(length > 0) | split("\t") + | {context: .[0], app_id: (.[1]|tonumber), verdict: .[2]}))}' "$ROWS" >"$JSON_OUT" 2>/dev/null || true + info "json report written to $JSON_OUT" + fi + rm -f "$pubb" "$required" "$ROWS" + [ "$unsatisfiable" -gt 0 ] && [ "$STRICT" = 1 ] && return 1 + return 0 +} + +self_test() { + local work pass=0 fail=0 + work="$(mktemp -d)" + mkdir -p "$work/repo/.github/workflows" "$work/cache" + cat >"$work/repo/.github/workflows/hypatia-scan.yml" <<'YML' +name: Hypatia Security Scan +on: [pull_request] +jobs: + hypatia: + uses: hyperpolymath/standards/.github/workflows/hypatia-scan-reusable.yml@cafebabe + secrets: inherit +YML + cat >"$work/repo/.github/workflows/other.yml" <<'YML' +name: Other +on: [pull_request] +jobs: + lake-build: + runs-on: ubuntu-latest + steps: [{run: "true"}] + analyze: + strategy: + matrix: + language: [actions, python] + runs-on: ubuntu-latest + steps: [{run: "true"}] +YML + cat >"$work/cache/hyperpolymath_standards_.github_workflows_hypatia-scan-reusable.yml_cafebabe" <<'YML' +name: Hypatia Reusable Scan +on: [workflow_call] +jobs: + scan: + name: Hypatia Neurosymbolic Analysis + runs-on: ubuntu-latest +YML + check() { # label expected-exit contexts-json strict + local label="$1" want="$2" file="$3" strict="${4:-0}" out status + out="$(REUSABLE_CACHE_DIR="$work/cache" REQUIRED_CONTEXTS_JSON="$file" \ + REQUIRED_CONTEXTS_STRICT="$strict" GITHUB_REPOSITORY=example/repo \ + bash "$0" "$work/repo" --no-network 2>&1)"; status=$? + if [ "$status" = "$want" ]; then pass=$((pass+1)); say "PASS $label" + else fail=$((fail+1)); say "FAIL $label — expected exit $want, got $status"; say "$out" | head -6; fi + } + printf '[{"context":"hypatia / Hypatia Neurosymbolic Analysis","app_id":15368,"source":"ruleset"},{"context":"lake-build","app_id":15368,"source":"ruleset"},{"context":"analyze (actions, python)","app_id":15368,"source":"ruleset"},{"context":"CodeRabbit","app_id":9999,"source":"ruleset"}]' >"$work/ok.json" + printf '[{"context":"Hypatia Neurosymbolic Analysis","app_id":15368,"source":"ruleset"}]' >"$work/bare.json" + printf '[{"context":"Retired Check","app_id":0,"source":"ruleset"}]' >"$work/phantom.json" + printf '[{"context":"lake","app_id":15368,"source":"ruleset"}]' >"$work/substring.json" + check "publishes-what-is-required" 0 "$work/ok.json" + check "bare-reusable-name-is-unsatisfiable-advisory" 0 "$work/bare.json" 0 + check "bare-reusable-name-is-unsatisfiable-strict" 1 "$work/bare.json" 1 + check "unknown-check-is-unsatisfiable-strict" 1 "$work/phantom.json" 1 + check "prefix-of-a-real-name-is-not-a-match" 1 "$work/substring.json" 1 + rm -rf "$work" + say "" + say "self-test: $pass passed, $fail failed" + [ "$fail" -eq 0 ] +} + +if [ "$SELF_TEST" = 1 ]; then self_test; exit $?; fi +run_audit diff --git a/scripts/tests/check-required-contexts-test.sh b/scripts/tests/check-required-contexts-test.sh new file mode 100755 index 000000000..cbd70d70c --- /dev/null +++ b/scripts/tests/check-required-contexts-test.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# +# check-required-contexts-test.sh — fixture suite for +# scripts/check-required-contexts.sh, the unsatisfiable-required-context gate. +# +# The script carries its own hermetic fixture: a synthetic repository (a +# reusable-wrapper caller, a plain job and a matrix job), a stubbed reusable in a +# cache directory, and required-context sets supplied as JSON. No network, no +# `gh`, no token, no API — so this runs in every environment and cannot be +# green-while-broken because a credential was missing. +# +# Branches driven: +# * a context the wrapper publishes -> satisfiable, exit 0 +# * the reusable's bare inner name (the #17 shape) -> unsatisfiable, advisory +# exit 0, strict exit 1 +# * an unknown name bound to no app integration -> strict exit 1 +# * a name that is only a PREFIX of a real one -> strict exit 1 +# (guards against the substring-matching bug this suite was written to catch) +# +# Run: bash scripts/tests/check-required-contexts-test.sh +set -uo pipefail +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +GATE="$SCRIPT_DIR/../check-required-contexts.sh" + +if [ ! -f "$GATE" ]; then + echo "FAIL: gate not found at $GATE" + exit 1 +fi + +if output="$(bash "$GATE" --self-test 2>&1)"; then + printf '%s\n' "$output" + echo "PASS $0" + exit 0 +fi + +printf '%s\n' "$output" +echo "::error file=$GATE::fixture suite failed" +exit 1