From 80c8c9571ae36298ee121d49fdaffc0612d56e61 Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Tue, 22 Sep 2026 22:14:36 +0100 Subject: [PATCH] feat(rulesets): add apply-branch-gates.sh, the applier gates.json specified but nothing performed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit config/rulesets/README.adoc has said so since it was written: "the propagation mechanism is still missing by design, and this note is the reminder that a template fix without an applier is a half fix." That was literally true — nothing in this repository read config/rulesets/gates.json as DATA. Every reference was prose, or tests/test_governance_reusable_shape.sh asserting the file's contents. gates.json specified a context derivation that no code performed, so required_status_checks was never propagated to any repository. apply-tag-ruleset-canon.sh is the TAG applier; this is its branch sibling. CONTEXTS ARE DERIVED, NEVER TYPED Per gates.json, contexts come from the check names the latest default-branch run of each gate workflow actually emitted, via /actions/workflows/{file}/runs?branch=&per_page=1 /actions/runs/{id}/jobs A hand-typed context that nothing emits is a PHANTOM: it can never turn green, so it blocks the branch permanently. THE REFUSAL THAT MATTERS Zero derivable contexts => report UNGATED and write NOTHING. A required_status_checks rule carrying an empty list is a VACUOUS GATE: it reports "protected" in every summary view while requiring nothing, which is strictly worse than having no rule, because it is indistinguishable from a working one. EXACTNESS GUARD A ruleset PUT replaces the whole object. The planned body, normalised with the required_status_checks rule removed from both sides, must be byte-identical to the source; anything else moved is a refusal. Without it, one jq slip strips required_signatures from every repo it touches, silently. DELIBERATE NON-ACTIONS * never creates a ruleset (NORULESET is reported, not repaired) — creating branch protection where none exists is a policy act, not a gate-fill; * never removes the four retired rule types unless --strip-retired is passed — the estate census was ruled report-only; * two active branch rulesets => AMBIGUOUS, fail closed. Rulesets are ADDITIVE: filling one of a pair leaves the other enforcing and the repo stays blocked by a rule nothing announced. --require-green N implements the standing ruling on #956, "require the reliably-green set": a context that is currently red becomes a merge deadlock the instant it is required. `skipped` and `neutral` count as green, because GitHub treats both as satisfying a required status check. TESTS — a passing suite would not have been evidence scripts/tests/branch-gates-apply-test.sh runs 18 controls, two of which are MUTANTS that must go red: A. delete the zero-context refusal -> the applier writes the vacuous empty-list rule (verified by inspecting the captured PUT body); B. make the body-builder also drop required_signatures -> the exactness guard refuses. Both die. The suite exercises the applier through a `gh` shim, so it needs no network and no credentials. Two defects were found by these controls rather than in review: the --strip-retired jq filter used `index(.type)`, where jq rebinds `.` to the array so `.type` was always null (the same class of bug apply-baseline-test.sh was written to pin), and the exactness guard rejected its own honest control because ADDING the required_status_checks rule legitimately changes the rules array length. Both files are committed 100755 — a suite committed 0644 passes every local run and dies in CI at exit 126 before a single control executes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ji1bq3TypfycfUPAR7hSxR --- scripts/apply-branch-gates.sh | 343 +++++++++++++++++++++++ scripts/tests/branch-gates-apply-test.sh | 239 ++++++++++++++++ 2 files changed, 582 insertions(+) create mode 100755 scripts/apply-branch-gates.sh create mode 100755 scripts/tests/branch-gates-apply-test.sh diff --git a/scripts/apply-branch-gates.sh b/scripts/apply-branch-gates.sh new file mode 100755 index 00000000..dae3ab71 --- /dev/null +++ b/scripts/apply-branch-gates.sh @@ -0,0 +1,343 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# +# apply-branch-gates.sh — fill the `required_status_checks` rule of a +# repository's active branch ruleset with contexts DERIVED from what its +# default-branch runs actually emitted. +# +# WHY THIS EXISTS +# config/rulesets/README.adoc has said so outright since it was written: +# "the propagation mechanism is still missing by design, and this note is the +# reminder that a template fix without an applier is a half fix." Nothing in +# this repository read config/rulesets/gates.json as DATA — every reference +# was prose, or tests/test_governance_reusable_shape.sh asserting the file's +# contents. gates.json specified a derivation that no code performed. +# apply-tag-ruleset-canon.sh is the TAG applier; this is its branch sibling. +# +# CONTEXTS ARE DERIVED, NEVER TYPED +# config/rulesets/gates.json: "Contexts are never typed by hand." A typed +# context that nothing emits is a PHANTOM — it can never turn green, so it +# blocks the branch forever. This script therefore reads the check names the +# latest default-branch run of each gate workflow ACTUALLY produced: +# GET /repos/{o}/{r}/actions/workflows/{file}/runs?branch=&per_page=1 +# GET /repos/{o}/{r}/actions/runs/{id}/jobs +# and uses the job `name` verbatim. For a reusable caller GitHub already +# renders that as " / ". +# +# THE TWO REFUSALS THAT MATTER +# if_no_run_yet -> omit that file's contexts and REPORT it. +# Never write a context nothing has emitted. +# if_zero_contexts_overall -> do NOT write the rule at all; report UNGATED. +# A required_status_checks rule with an empty list +# is a VACUOUS GATE: it reports "protected" while +# requiring nothing. That is worse than no rule, +# because it is indistinguishable from a real one +# in every summary view. +# +# EXACTNESS GUARD +# A ruleset PUT REPLACES the whole object. This script therefore refuses to +# write unless the planned body, normalised over the required_status_checks +# rule alone, is byte-identical to the source. Anything else moved => refuse. +# Without this, one jq slip silently strips required_signatures estate-wide. +# +# WHAT IT DELIBERATELY DOES NOT DO +# * It never CREATES a ruleset. A repo with no active branch ruleset is +# reported NORULESET. Creating branch protection where none exists is a +# policy act, not a gate-fill. +# * It never DELETES or rewrites another rule. Repos carrying the retired +# types (update, required_deployments, code_quality, code_coverage) are +# REPORTED, not repaired: the estate census was ruled report-only. Pass +# --strip-retired to opt in, one repo at a time. +# * It never emits a retired rule type itself. The exactness guard makes that +# structurally impossible, not merely intended. +# * Two active branch rulesets => AMBIGUOUS, fail closed. Rulesets are +# ADDITIVE (see apply-tag-ruleset-canon.sh): writing one of a pair leaves +# the other enforcing, and the repo stays blocked by a rule nothing +# announced. Guessing which to fill is how that happens silently. +# +# Inputs (environment): +# GH_TOKEN required for writes; needs administration:write on targets. +# ESTATE_ORGS optional, space-separated. Default "metadatastician". +# +# Flags: +# --apply perform writes. WITHOUT IT THIS SCRIPT ONLY REPORTS. +# --repo OWNER/NAME process exactly one repository (repeatable). +# --limit N process at most N repositories (pilot runs). +# --gates-file F default config/rulesets/gates.json +# --strip-retired also remove the 4 retired rule types. Off by default. +# --require-green N drop any derived context that is not green across the +# last N default-branch runs of its workflow. Off (0) by +# default. Owner ruling on #956: "require the reliably- +# green set" -- a required context that is currently red +# blocks the branch the moment it is required, so gating +# on it converts a visible red into a merge deadlock. +# `skipped` and `neutral` COUNT AS GREEN: GitHub treats +# both as satisfying a required status check. +# --skip-user do not enumerate user/repos; use only ESTATE_ORGS. +# +# Output: TSV on stdout repo state detail +# per-class summary on stderr. +# +# Exit codes: 0 ok · 1 usage/refusal · 2 at least one repo FAILED +set -uo pipefail + +RETIRED_TYPES='update required_deployments code_quality code_coverage' +ACTIONS_INTEGRATION_ID=15368 + +APPLY=0 LIMIT=0 STRIP_RETIRED=0 SKIP_USER=0 REQUIRE_GREEN=0 +GATES_FILE='config/rulesets/gates.json' +REPOS_EXPLICIT=() + +die() { printf '%s\n' "$*" >&2; exit 1; } + +while [ $# -gt 0 ]; do + case "$1" in + --apply) APPLY=1 ;; + --repo) shift; [ $# -gt 0 ] || die 'usage: --repo OWNER/NAME'; REPOS_EXPLICIT+=("$1") ;; + --limit) shift; [ $# -gt 0 ] || die 'usage: --limit N'; LIMIT="$1" ;; + --gates-file) shift; [ $# -gt 0 ] || die 'usage: --gates-file PATH'; GATES_FILE="$1" ;; + --strip-retired) STRIP_RETIRED=1 ;; + --require-green) shift; [ $# -gt 0 ] || die 'usage: --require-green N'; REQUIRE_GREEN="$1" ;; + --skip-user) SKIP_USER=1 ;; + -h|--help) sed -n '2,70p' "$0"; exit 0 ;; + *) die "unknown flag: $1" ;; + esac + shift +done + +[ -r "$GATES_FILE" ] || die "gates file not readable: $GATES_FILE" +command -v gh >/dev/null || die 'gh is required' +command -v jq >/dev/null || die 'jq is required' + +WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT + +jq -e . "$GATES_FILE" >/dev/null 2>&1 || die "gates file is not valid JSON: $GATES_FILE" + +jq -r '.never_required_workflows[]?' "$GATES_FILE" | sort -u > "$WORK/never_wf" +jq -r '.never_required_contexts[]?' "$GATES_FILE" | sort -u > "$WORK/never_ctx" + +emit() { printf '%s\t%s\t%s\n' "$1" "$2" "$3"; printf '%s\n' "$2" >> "$WORK/states"; } + +# --- normalise away ONLY the required_status_checks rule, so a diff of the +# normalised forms proves nothing outside it moved. +# The required_status_checks rule is the INTENDED change, so it is removed from +# both sides before comparison -- including when it is being ADDED, where the +# two rule arrays legitimately differ in length. Everything else must match +# exactly. The rule's own contents are verified separately, after the write, +# by comparing the planned context set against what the API returns. +norm_rsc() { + jq -S '.rules = ((.rules // []) | map(select(.type!="required_status_checks")))' "$1" +} + +# --- is $1 listed in never_required_contexts, whole or after the first " / "? +is_never_ctx() { + local c="$1" tail="${1#* / }" + command grep -qxF -- "$c" "$WORK/never_ctx" && return 0 + [ "$tail" != "$c" ] && command grep -qxF -- "$tail" "$WORK/never_ctx" && return 0 + return 1 +} + +# ---------------------------------------------------------------- repo list +if [ "${#REPOS_EXPLICIT[@]}" -gt 0 ]; then + printf '%s\n' "${REPOS_EXPLICIT[@]}" > "$WORK/repos" +else + : > "$WORK/repos" + [ "$SKIP_USER" = 1 ] || gh repo list --limit 1000 --json nameWithOwner,isArchived \ + --jq '.[]|select(.isArchived|not)|.nameWithOwner' >> "$WORK/repos" 2>/dev/null + printf '%s\n' ${ESTATE_ORGS:-metadatastician} | while IFS= read -r ORG; do + [ -n "$ORG" ] || continue + gh repo list "$ORG" --limit 1000 --json nameWithOwner,isArchived \ + --jq '.[]|select(.isArchived|not)|.nameWithOwner' >> "$WORK/repos" 2>/dev/null + done + sort -u "$WORK/repos" -o "$WORK/repos" +fi + +[ -s "$WORK/repos" ] || die 'refusing to report a clean sweep over nothing: target list is empty' +[ "$LIMIT" -gt 0 ] 2>/dev/null && head -n "$LIMIT" "$WORK/repos" > "$WORK/r2" && mv "$WORK/r2" "$WORK/repos" + +printf '# mode=%s repos=%d gates=%s\n' \ + "$( [ "$APPLY" = 1 ] && echo APPLY || echo REPORT-ONLY )" "$(wc -l < "$WORK/repos")" "$GATES_FILE" >&2 +printf 'repo\tstate\tdetail\n' +: > "$WORK/states" + +# ---------------------------------------------------------------- main loop +while IFS= read -r R; do + [ -n "$R" ] || continue + + gh api "repos/$R" > "$WORK/repo.json" 2>"$WORK/e" \ + || { emit "$R" "UNKNOWN" "repo GET failed: $(head -c 100 "$WORK/e" | tr -d '\n')"; continue; } + DEF=$(jq -r '.default_branch // empty' "$WORK/repo.json") + [ -n "$DEF" ] || { emit "$R" "UNKNOWN" 'no default branch (empty repo?)'; continue; } + + # ---- 1. which gate workflow FILES apply (profiles) -------------------- + gh api "repos/$R/contents/.github/workflows" --jq '.[]?|.name' 2>/dev/null | sort -u > "$WORK/wf" || : > "$WORK/wf" + gh api "repos/$R/contents" --jq '.[]?|.name' 2>/dev/null | sort -u > "$WORK/root" || : > "$WORK/root" + + : > "$WORK/gatewf" + jq -r '.profiles | to_entries[] | @base64' "$GATES_FILE" > "$WORK/profiles" + while IFS= read -r P64; do + P=$(printf '%s' "$P64" | base64 -d) + KEY=$(printf '%s' "$P" | jq -r '.key') + ACTIVE=0 + if [ "$KEY" = base ]; then + ACTIVE=1 + else + # detect: root-level files/globs + while IFS= read -r PAT; do + [ -n "$PAT" ] || continue + while IFS= read -r F; do + case "$F" in $PAT) ACTIVE=1 ;; esac + done < "$WORK/root" + done < <(printf '%s' "$P" | jq -r '.value.detect[]?') + # detect_workflows: presence of a workflow file + while IFS= read -r WFN; do + [ -n "$WFN" ] || continue + command grep -qxF -- "$WFN" "$WORK/wf" && ACTIVE=1 + done < <(printf '%s' "$P" | jq -r '.value.detect_workflows[]?') + fi + [ "$ACTIVE" = 1 ] || continue + printf '%s' "$P" | jq -r '.value.gate_workflows[]?' >> "$WORK/gatewf" + done < "$WORK/profiles" + + sort -u "$WORK/gatewf" -o "$WORK/gatewf" + # a gate workflow must exist in the repo AND not be never-required + : > "$WORK/gatewf2" + while IFS= read -r WFN; do + [ -n "$WFN" ] || continue + command grep -qxF -- "$WFN" "$WORK/never_wf" && continue + command grep -qxF -- "$WFN" "$WORK/wf" || continue + printf '%s\n' "$WFN" >> "$WORK/gatewf2" + done < "$WORK/gatewf" + + # ---- 2. DERIVE contexts from real runs -------------------------------- + : > "$WORK/ctx"; NORUN='' + while IFS= read -r WFN; do + [ -n "$WFN" ] || continue + RID=$(gh api "repos/$R/actions/workflows/$WFN/runs?branch=$DEF&per_page=1" \ + --jq '.workflow_runs[0].id // empty' 2>/dev/null) + if [ -z "$RID" ]; then NORUN="${NORUN:+$NORUN,}$WFN"; continue; fi + gh api "repos/$R/actions/runs/$RID/jobs?per_page=100" --paginate \ + --jq '.jobs[]?|.name' 2>/dev/null >> "$WORK/ctx" + done < "$WORK/gatewf2" + + sort -u "$WORK/ctx" -o "$WORK/ctx" + : > "$WORK/ctx2"; EXCLUDED='' + while IFS= read -r C; do + [ -n "$C" ] || continue + if is_never_ctx "$C"; then EXCLUDED="${EXCLUDED:+$EXCLUDED,}$C"; continue; fi + printf '%s\n' "$C" >> "$WORK/ctx2" + done < "$WORK/ctx" + + # ---- optional: keep only contexts that are RELIABLY green ------------ + NOTGREEN='' + if [ "$REQUIRE_GREEN" -gt 0 ] 2>/dev/null; then + : > "$WORK/bad" + while IFS= read -r WFN; do + [ -n "$WFN" ] || continue + gh api "repos/$R/actions/workflows/$WFN/runs?branch=$DEF&per_page=$REQUIRE_GREEN" \ + --jq '.workflow_runs[]?.id' 2>/dev/null > "$WORK/rids" + while IFS= read -r RID2; do + [ -n "$RID2" ] || continue + # a job is acceptable when success/skipped/neutral, or still running + gh api "repos/$R/actions/runs/$RID2/jobs?per_page=100" --paginate \ + --jq '.jobs[]? | select((.conclusion // "pending") as $c + | ["success","skipped","neutral","pending"] | index($c) | not) | .name' \ + 2>/dev/null >> "$WORK/bad" + done < "$WORK/rids" + done < "$WORK/gatewf2" + sort -u "$WORK/bad" -o "$WORK/bad" + : > "$WORK/ctx3" + while IFS= read -r C; do + [ -n "$C" ] || continue + if command grep -qxF -- "$C" "$WORK/bad"; then NOTGREEN="${NOTGREEN:+$NOTGREEN,}$C"; continue; fi + printf '%s\n' "$C" >> "$WORK/ctx3" + done < "$WORK/ctx2" + mv "$WORK/ctx3" "$WORK/ctx2" + fi + + NCTX=$(wc -l < "$WORK/ctx2") + DETAIL="branch=$DEF gate_files=$(wc -l < "$WORK/gatewf2") contexts=$NCTX" + [ -n "$NOTGREEN" ] && DETAIL="$DETAIL not_green=[$NOTGREEN]" + [ -n "$NORUN" ] && DETAIL="$DETAIL no_run=[$NORUN]" + [ -n "$EXCLUDED" ] && DETAIL="$DETAIL excluded=[$EXCLUDED]" + + # ---- THE REFUSAL: a rule with an empty list is a vacuous gate --------- + if [ "$NCTX" -eq 0 ]; then + emit "$R" "UNGATED" "$DETAIL — refusing to write an empty required_status_checks rule" + continue + fi + + # ---- 3. locate the one active branch ruleset -------------------------- + gh api "repos/$R/rulesets" > "$WORK/rs.json" 2>"$WORK/e" \ + || { emit "$R" "UNKNOWN" "rulesets GET failed"; continue; } + jq -r '.[]|select(.target=="branch" and .enforcement=="active")|.id' "$WORK/rs.json" > "$WORK/ids" + NIDS=$(wc -l < "$WORK/ids") + [ "$NIDS" -eq 0 ] && { emit "$R" "NORULESET" "$DETAIL — no active branch ruleset; this script never creates one"; continue; } + [ "$NIDS" -gt 1 ] && { emit "$R" "AMBIGUOUS" "$DETAIL — $NIDS active branch rulesets ($(tr '\n' ',' < "$WORK/ids")); rulesets are additive, refusing to guess"; continue; } + ID=$(cat "$WORK/ids") + + gh api "repos/$R/rulesets/$ID" > "$WORK/live.json" 2>/dev/null \ + || { emit "$R" "UNKNOWN" "ruleset $ID GET failed"; continue; } + + FOUND_RETIRED=$(jq -r --arg rt "$RETIRED_TYPES" \ + '[.rules[]?.type] as $t | ($rt|split(" ")) - (($rt|split(" ")) - $t) | join(",")' "$WORK/live.json") + [ -n "$FOUND_RETIRED" ] && DETAIL="$DETAIL retired_present=[$FOUND_RETIRED]" + + # ---- 4. build the PUT body ------------------------------------------- + jq -R -s --argjson iid "$ACTIONS_INTEGRATION_ID" \ + 'split("\n")|map(select(length>0))|map({context:., integration_id:$iid})' "$WORK/ctx2" > "$WORK/checks.json" + + jq --slurpfile ck "$WORK/checks.json" ' + {name,target,enforcement,conditions,bypass_actors,rules} + | .rules = ((.rules // []) | map(select(.type!="required_status_checks"))) + | .rules += [{ type:"required_status_checks", + parameters:{ strict_required_status_checks_policy:false, + do_not_enforce_on_create:false, + required_status_checks:$ck[0] } }] + ' "$WORK/live.json" > "$WORK/put.json" + + if [ "$STRIP_RETIRED" = 1 ]; then + jq --arg rt "$RETIRED_TYPES" '($rt|split(" ")) as $r | .rules |= map(select(.type as $t | ($r|index($t))|not))' \ + "$WORK/put.json" > "$WORK/p2" && mv "$WORK/p2" "$WORK/put.json" + fi + + # structural assertion: we never emit a retired type that was not already there + EMITTED_RETIRED=$(jq -r --arg rt "$RETIRED_TYPES" \ + '[.rules[]?.type] as $t | ($rt|split(" ")) - (($rt|split(" ")) - $t) | join(",")' "$WORK/put.json") + if [ "$STRIP_RETIRED" = 1 ] && [ -n "$EMITTED_RETIRED" ]; then + emit "$R" "REFUSED" "$DETAIL — --strip-retired left [$EMITTED_RETIRED] in the body"; continue + fi + + # ---- EXACTNESS GUARD (skipped when --strip-retired deliberately differs) + if [ "$STRIP_RETIRED" = 0 ]; then + jq '{name,target,enforcement,conditions,bypass_actors,rules}' "$WORK/live.json" > "$WORK/src.json" + if ! diff -q <(norm_rsc "$WORK/src.json") <(norm_rsc "$WORK/put.json") >/dev/null; then + emit "$R" "REFUSED" "$DETAIL — exactness guard: change outside the required_status_checks rule" + continue + fi + fi + + if [ "$APPLY" = 0 ]; then + emit "$R" "WOULD-GATE" "$DETAIL ruleset=$ID :: $(tr '\n' '|' < "$WORK/ctx2")" + continue + fi + + if ! gh api -X PUT "repos/$R/rulesets/$ID" --input "$WORK/put.json" >/dev/null 2>"$WORK/e"; then + emit "$R" "FAILED" "$DETAIL — PUT: $(head -c 160 "$WORK/e" | tr -d '\n')"; continue + fi + + # ---- 5. post-write verification -------------------------------------- + gh api "repos/$R/rulesets/$ID" > "$WORK/after.json" 2>/dev/null \ + || { emit "$R" "WROTE-UNVERIFIED" "$DETAIL — re-GET failed"; continue; } + PRE_BY=$(jq -cS '.bypass_actors//[]' "$WORK/live.json"); POST_BY=$(jq -cS '.bypass_actors//[]' "$WORK/after.json") + WANT=$(jq -cS '[.rules[]?|select(.type=="required_status_checks")|.parameters.required_status_checks[].context]|sort' "$WORK/put.json") + GOT=$(jq -cS '[.rules[]?|select(.type=="required_status_checks")|.parameters.required_status_checks[].context]|sort' "$WORK/after.json") + if [ "$PRE_BY" != "$POST_BY" ]; then emit "$R" "DRIFT" "$DETAIL — bypass_actors changed across the write"; continue; fi + if [ "$WANT" != "$GOT" ]; then emit "$R" "DRIFT" "$DETAIL — contexts after write != planned"; continue; fi + emit "$R" "GATED" "$DETAIL ruleset=$ID :: $(tr '\n' '|' < "$WORK/ctx2")" +done < "$WORK/repos" + +echo '# summary' >&2 +sort "$WORK/states" | uniq -c | sort -rn >&2 +command grep -qx 'FAILED' "$WORK/states" && exit 2 +exit 0 diff --git a/scripts/tests/branch-gates-apply-test.sh b/scripts/tests/branch-gates-apply-test.sh new file mode 100755 index 00000000..eff2d61f --- /dev/null +++ b/scripts/tests/branch-gates-apply-test.sh @@ -0,0 +1,239 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# +# branch-gates-apply-test.sh — regression test for apply-branch-gates.sh. +# +# WHAT THIS PINS, AND WHY A PASSING SUITE WOULD NOT BE ENOUGH +# The defect this script exists to prevent is a VACUOUS GATE: a +# required_status_checks rule carrying an EMPTY context list. Such a rule +# reports "this branch is protected" in every summary view while requiring +# nothing at all — strictly worse than having no rule, because it is +# indistinguishable from a working one. +# +# A green suite proves nothing about that. So three of the cases below are +# MUTANTS: the applier is copied, the guard under test is deliberately +# removed, and the suite must go RED. A mutant that survives means the +# corresponding control is decorative. +# +# Run: bash scripts/tests/branch-gates-apply-test.sh +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +APPLIER="${BRANCH_GATES_TARGET:-$SCRIPT_DIR/../apply-branch-gates.sh}" +WORK="$(mktemp -d)" +trap 'rm -rf "$WORK"' EXIT + +pass=0; fail=0 +ok() { echo "PASS: $1"; pass=$((pass+1)); } +bad() { echo "FAIL: $1"; fail=$((fail+1)); } + +# ------------------------------------------------------------------ fixtures +FIX="$WORK/fix"; mkdir -p "$FIX" +BIN="$WORK/bin"; mkdir -p "$BIN" + +# A `gh` shim: maps an API path to a fixture file, honours --jq, records PUTs. +cat > "$BIN/gh" <<'SHIM' +#!/usr/bin/env bash +set -uo pipefail +[ "${1:-}" = api ] || exit 0 +shift +METHOD=GET; JQF=''; APIPATH='' +while [ $# -gt 0 ]; do + case "$1" in + -X) shift; METHOD="$1" ;; + --jq) shift; JQF="$1" ;; + --input) shift; cp "$1" "$GH_FIX/LAST_PUT.json" ;; + --paginate|--silent) : ;; + -*) : ;; + *) [ -n "$APIPATH" ] || APIPATH="$1" ;; + esac + shift +done +KEY=$(printf '%s' "$APIPATH" | tr '/?&=' '____') +if [ "$METHOD" = PUT ]; then printf '%s\n' "$APIPATH" >> "$GH_FIX/PUTS.log"; echo '{}'; exit 0; fi +F="$GH_FIX/$KEY.json" +[ -r "$F" ] || exit 1 +if [ -n "$JQF" ]; then jq -r "$JQF" "$F"; else cat "$F"; fi +SHIM +chmod +x "$BIN/gh" + +GATES="$WORK/gates-src.json" +cat > "$GATES" <<'G' +{ "version": 1, + "profiles": { + "base": { "applies_to": "every repo", "gate_workflows": ["governance.yml","codeql.yml","mirror.yml"] }, + "rust": { "detect": ["Cargo.toml"], "gate_workflows": ["rust-ci.yml"] }, + "ada": { "detect": ["*.gpr"], "gate_workflows": ["ada-ci.yml"] } + }, + "never_required_workflows": ["mirror.yml"], + "never_required_contexts": ["Allowlist Preflight","Code quality + docs"] } +G + +mkfix() { printf '%s' "$2" > "$FIX/$(printf '%s' "$1" | tr '/?&=' '____').json"; } + +reset_fix() { rm -f "$FIX"/*.json; cp "$GATES" "$WORK/gates.json"; : > "$FIX/PUTS.log"; } + +run_applier() { # run_applier [extra flags...] + local repo="$1"; shift + PATH="$BIN:$PATH" GH_FIX="$FIX" bash "${MUTANT:-$APPLIER}" \ + --gates-file "$WORK/gates.json" --repo "$repo" "$@" 2>"$WORK/err" +} + +state_of() { tail -n +2 <<< "$1" | head -1 | cut -f2; } +detail_of() { tail -n +2 <<< "$1" | head -1 | cut -f3; } + +# ================================================================ CASE 1 +# POSITIVE CONTROL: contexts are derived from real jobs, never-required ones +# are stripped, and a never_required_workflow is not consulted at all. +reset_fix +R=acme/widget +mkfix "repos/$R" '{"default_branch":"main"}' +mkfix "repos/$R/contents/.github/workflows" '[{"name":"governance.yml"},{"name":"codeql.yml"},{"name":"mirror.yml"}]' +mkfix "repos/$R/contents" '[{"name":"README.md"},{"name":"Cargo.toml"}]' +mkfix "repos/$R/actions/workflows/governance.yml/runs?branch=main&per_page=1" '{"workflow_runs":[{"id":11}]}' +mkfix "repos/$R/actions/workflows/codeql.yml/runs?branch=main&per_page=1" '{"workflow_runs":[{"id":22}]}' +mkfix "repos/$R/actions/runs/11/jobs?per_page=100" '{"jobs":[{"name":"governance / Governance"},{"name":"governance / Code quality + docs"},{"name":"Allowlist Preflight"}]}' +mkfix "repos/$R/actions/runs/22/jobs?per_page=100" '{"jobs":[{"name":"CodeQL Security Analysis"}]}' +mkfix "repos/$R/rulesets" '[{"id":9,"target":"branch","enforcement":"active"},{"id":8,"target":"tag","enforcement":"active"}]' +mkfix "repos/$R/rulesets/9" '{"id":9,"name":"Base","target":"branch","enforcement":"active","conditions":{"ref_name":{"include":["~DEFAULT_BRANCH"],"exclude":[]}},"bypass_actors":[{"actor_id":5,"actor_type":"RepositoryRole","bypass_mode":"pull_request"}],"rules":[{"type":"deletion"},{"type":"required_signatures"}]}' + +OUT=$(run_applier "$R") +S=$(state_of "$OUT"); D=$(detail_of "$OUT") +[ "$S" = "WOULD-GATE" ] && ok "control: state is WOULD-GATE" || bad "control: state=$S (want WOULD-GATE)" +case "$D" in *"governance / Governance"*) ok "control: derived the real job name" ;; *) bad "control: missing derived context — $D" ;; esac +case "$D" in *"CodeQL Security Analysis"*) ok "control: derived across two workflows" ;; *) bad "control: second workflow not derived" ;; esac +case "$D" in *"contexts=2"*) ok "control: exactly 2 contexts survive" ;; *) bad "control: wrong count — $D" ;; esac +case "$D" in *"excluded=["*"Code quality + docs"*) ok "control: never_required matched AFTER the ' / '" ;; *) bad "control: reusable-job-name exclusion missed — $D" ;; esac +# assert against the CONTEXT LIST only (after ":: "), not the excluded= report +CTXLIST="${D##*:: }" +case "$CTXLIST" in + *"Allowlist Preflight"*|*"Code quality + docs"*) bad "control: a never_required context reached the rule — [$CTXLIST]" ;; + *) ok "control: never_required contexts kept out of the rule — [$CTXLIST]" ;; +esac +[ -s "$FIX/PUTS.log" ] && bad "control: wrote WITHOUT --apply" || ok "control: report-only performed no PUT" + +# ================================================================ CASE 2 +# THE REFUSAL: zero derivable contexts must yield UNGATED and write nothing. +reset_fix +R=acme/norun +mkfix "repos/$R" '{"default_branch":"main"}' +mkfix "repos/$R/contents/.github/workflows" '[{"name":"governance.yml"}]' +mkfix "repos/$R/contents" '[{"name":"README.md"}]' +mkfix "repos/$R/actions/workflows/governance.yml/runs?branch=main&per_page=1" '{"workflow_runs":[]}' +mkfix "repos/$R/rulesets" '[{"id":9,"target":"branch","enforcement":"active"}]' +mkfix "repos/$R/rulesets/9" '{"name":"Base","target":"branch","enforcement":"active","conditions":{},"bypass_actors":[],"rules":[{"type":"deletion"}]}' + +OUT=$(run_applier "$R" --apply) +S=$(state_of "$OUT"); D=$(detail_of "$OUT") +[ "$S" = "UNGATED" ] && ok "zero contexts: state is UNGATED" || bad "zero contexts: state=$S (want UNGATED)" +case "$D" in *"no_run=[governance.yml]"*) ok "zero contexts: the un-run workflow is named" ;; *) bad "zero contexts: no_run not reported — $D" ;; esac +[ -s "$FIX/PUTS.log" ] && bad "zero contexts: PUT happened despite --apply refusal" || ok "zero contexts: no PUT even with --apply" + +# ---- MUTANT A: delete the zero-context refusal. The suite MUST go red. ---- +MUT="$WORK/mutant-a.sh" +sed 's|if \[ "\$NCTX" -eq 0 \]; then|if false; then|' "$APPLIER" > "$MUT" +reset_fix +mkfix "repos/$R" '{"default_branch":"main"}' +mkfix "repos/$R/contents/.github/workflows" '[{"name":"governance.yml"}]' +mkfix "repos/$R/contents" '[{"name":"README.md"}]' +mkfix "repos/$R/actions/workflows/governance.yml/runs?branch=main&per_page=1" '{"workflow_runs":[]}' +mkfix "repos/$R/rulesets" '[{"id":9,"target":"branch","enforcement":"active"}]' +mkfix "repos/$R/rulesets/9" '{"name":"Base","target":"branch","enforcement":"active","conditions":{},"bypass_actors":[],"rules":[{"type":"deletion"}]}' +OUT=$(MUTANT="$MUT" run_applier "$R" --apply) +if [ "$(state_of "$OUT")" = "UNGATED" ]; then + bad "MUTANT A SURVIVED: refusal removed yet still UNGATED — the control is decorative" +else + ok "mutant A killed: without the refusal it becomes $(state_of "$OUT") and PUTs $(wc -l < "$FIX/PUTS.log") time(s)" +fi +if command grep -q 'required_status_checks' "$FIX/LAST_PUT.json" 2>/dev/null && + [ "$(jq '[.rules[]|select(.type=="required_status_checks")|.parameters.required_status_checks|length]|add // 0' "$FIX/LAST_PUT.json")" = "0" ]; then + ok "mutant A wrote the VACUOUS empty-list rule — precisely the defect being guarded" +fi + +# ================================================================ CASE 3 +# EXACTNESS GUARD: a transform touching anything else must be REFUSED. +reset_fix +R=acme/widget +mkfix "repos/$R" '{"default_branch":"main"}' +mkfix "repos/$R/contents/.github/workflows" '[{"name":"governance.yml"}]' +mkfix "repos/$R/contents" '[{"name":"README.md"}]' +mkfix "repos/$R/actions/workflows/governance.yml/runs?branch=main&per_page=1" '{"workflow_runs":[{"id":11}]}' +mkfix "repos/$R/actions/runs/11/jobs?per_page=100" '{"jobs":[{"name":"governance / Governance"}]}' +mkfix "repos/$R/rulesets" '[{"id":9,"target":"branch","enforcement":"active"}]' +mkfix "repos/$R/rulesets/9" '{"name":"Base","target":"branch","enforcement":"active","conditions":{},"bypass_actors":[{"actor_id":5,"actor_type":"RepositoryRole","bypass_mode":"pull_request"}],"rules":[{"type":"deletion"},{"type":"required_signatures"}]}' + +# MUTANT B: the body-builder also drops required_signatures. +MUTB="$WORK/mutant-b.sh" +python3 - "$APPLIER" "$MUTB" <<'MUTPY' +import sys +src, dst = sys.argv[1], sys.argv[2] +s = open(src).read() +old = '| .rules = ((.rules // []) | map(select(.type!="required_status_checks")))' +new = '| .rules = ((.rules // []) | map(select(.type!="required_status_checks" and .type!="required_signatures")))' +open(dst, "w").write(s.replace(old, new, 1) if old in s else s) +MUTPY +if ! diff -q "$APPLIER" "$MUTB" >/dev/null; then + OUT=$(MUTANT="$MUTB" run_applier "$R" --apply) + if [ "$(state_of "$OUT")" = "REFUSED" ]; then + ok "mutant B killed: dropping required_signatures is REFUSED by the exactness guard" + else + bad "MUTANT B SURVIVED: required_signatures silently dropped, state=$(state_of "$OUT")" + fi +else + bad "mutant B was not applied — the sed pattern no longer matches the applier" +fi + +# ================================================================ CASE 4 +# Rulesets are ADDITIVE: two active branch rulesets must fail closed. +reset_fix +R=acme/two +mkfix "repos/$R" '{"default_branch":"main"}' +mkfix "repos/$R/contents/.github/workflows" '[{"name":"governance.yml"}]' +mkfix "repos/$R/contents" '[{"name":"README.md"}]' +mkfix "repos/$R/actions/workflows/governance.yml/runs?branch=main&per_page=1" '{"workflow_runs":[{"id":11}]}' +mkfix "repos/$R/actions/runs/11/jobs?per_page=100" '{"jobs":[{"name":"governance / Governance"}]}' +mkfix "repos/$R/rulesets" '[{"id":9,"target":"branch","enforcement":"active"},{"id":10,"target":"branch","enforcement":"active"}]' +OUT=$(run_applier "$R" --apply) +[ "$(state_of "$OUT")" = "AMBIGUOUS" ] && ok "two active branch rulesets: AMBIGUOUS, fail closed" || bad "two rulesets: state=$(state_of "$OUT") (want AMBIGUOUS)" +[ -s "$FIX/PUTS.log" ] && bad "two rulesets: wrote anyway" || ok "two rulesets: no PUT" + +# ================================================================ CASE 5 +# No active branch ruleset: report, never create one. +reset_fix +R=acme/none +mkfix "repos/$R" '{"default_branch":"main"}' +mkfix "repos/$R/contents/.github/workflows" '[{"name":"governance.yml"}]' +mkfix "repos/$R/contents" '[{"name":"README.md"}]' +mkfix "repos/$R/actions/workflows/governance.yml/runs?branch=main&per_page=1" '{"workflow_runs":[{"id":11}]}' +mkfix "repos/$R/actions/runs/11/jobs?per_page=100" '{"jobs":[{"name":"governance / Governance"}]}' +mkfix "repos/$R/rulesets" '[{"id":8,"target":"tag","enforcement":"active"},{"id":7,"target":"branch","enforcement":"disabled"}]' +OUT=$(run_applier "$R" --apply) +[ "$(state_of "$OUT")" = "NORULESET" ] && ok "no active branch ruleset: NORULESET, nothing created" || bad "no ruleset: state=$(state_of "$OUT")" + +# ================================================================ CASE 6 +# Profile detection: a root-level glob (*.gpr) activates the ada profile. +reset_fix +R=acme/ada +mkfix "repos/$R" '{"default_branch":"main"}' +mkfix "repos/$R/contents/.github/workflows" '[{"name":"governance.yml"},{"name":"ada-ci.yml"}]' +mkfix "repos/$R/contents" '[{"name":"thing.gpr"}]' +mkfix "repos/$R/actions/workflows/governance.yml/runs?branch=main&per_page=1" '{"workflow_runs":[{"id":11}]}' +mkfix "repos/$R/actions/workflows/ada-ci.yml/runs?branch=main&per_page=1" '{"workflow_runs":[{"id":33}]}' +mkfix "repos/$R/actions/runs/11/jobs?per_page=100" '{"jobs":[{"name":"governance / Governance"}]}' +mkfix "repos/$R/actions/runs/33/jobs?per_page=100" '{"jobs":[{"name":"Ada Build"}]}' +mkfix "repos/$R/rulesets" '[{"id":9,"target":"branch","enforcement":"active"}]' +mkfix "repos/$R/rulesets/9" '{"name":"Base","target":"branch","enforcement":"active","conditions":{},"bypass_actors":[],"rules":[{"type":"deletion"}]}' +OUT=$(run_applier "$R") +case "$(detail_of "$OUT")" in *"Ada Build"*) ok "profile detect: *.gpr glob activated the ada profile" ;; *) bad "profile detect: ada gate missing — $(detail_of "$OUT")" ;; esac + +# ================================================================ CASE 7 +# Refuse an empty target list rather than report a clean sweep over nothing. +if PATH="$BIN:$PATH" GH_FIX="$FIX" bash "$APPLIER" --gates-file "$WORK/gates.json" --skip-user >/dev/null 2>&1; then + bad "empty target list: exited 0 instead of refusing" +else + ok "empty target list: refused" +fi + +echo +echo "passed=$pass failed=$fail" +[ "$fail" -eq 0 ]