Skip to content

The ruleset profile argued from a fourteen-commit repository that now has sixty-six - #70

Merged
ChelseaKR merged 1 commit into
mainfrom
docs/ruleset-evidence-remeasured
Sep 7, 2026
Merged

ChelseaKR merged 1 commit into
mainfrom
docs/ruleset-evidence-remeasured

Conversation

@ChelseaKR

Copy link
Copy Markdown
Owner

What was wrong

.github/rulesets/README.md is the document a person reads before deciding whether to
apply main.json. Two of its measurements were taken on 2026-08-15 and never re-read:

  • it argued for required_signatures from "all fourteen commits on main" — there are 66;
  • it argued that required_linear_history costs nothing given "the three merge commits
    already on main" — there are 12.

Both conclusions survive re-measurement. Neither piece of evidence did.

The worse finding

Re-measuring turned up an inverted reading. A local git log --format='%G?' over this
repository reports N — "no signature" — for thirteen commits on main. Those commits
are not unsigned.
They are SSH-signed with the owner's own key, and git 2.55 classifies
an SSH signature as N when no gpg.ssh.allowedSignersFile is configured; supply any
allowed-signers file and the same commit reports U (good signature, untrusted key). The
other fifty-three are PGP-signed by GitHub's web-flow key and report E, key not in the
local keyring.

Measured against the API the document actually cites:

gh api 'repos/ChelseaKR/perimeter/commits?sha=main&per_page=100' \
  --jq '[.[] | select(.commit.verification.verified | not)] | length'
0

Zero commits on main carry no signature; all 66 verify. That API verification is also
what GitHub's own required_signatures rule enforces, so it is the instrument that
matches the rule. An audit reading N as "unsigned" would report thirteen unsigned
commits that do not exist — a failed read published as a measurement, which is this
portfolio's most common defect, produced here by a check looking for it. The document now
names the instrument, the trap, and the one-call measurement.

What changed

  • .github/rulesets/README.md: both counts corrected and stated against the commit they
    were measured at
    (d4f533f, 2026-09-06), so main advancing does not falsify them
    and no hand edit is owed per merge; the %G? trap recorded; the "no ruleset applied"
    table re-confirmed today.
  • tests/test_ruleset_evidence.py: new.
    • Commit count and merge count re-measured from the SHA the prose pins. Because git
      history is immutable, this is not a hand-maintained counter that every merge
      re-jams. It fails on a number edited without moving the pin, a pin advanced without
      re-measuring, a claimed measurement date earlier than the commit it claims to read,
      or a checkout too shallow to see that commit.
    • Every required status check context in main.json derived from the workflow files,
      so a renamed job fails here rather than silently emptying the profile on the day it
      is applied — which is the failure mode the document itself warns about.
  • CHANGELOG.md: entry under [Unreleased].

How it was verified

807 passed (full suite less the two Node/browser a11y gates, which need npm ci and
Playwright); ruff check, ruff format --check clean.

Each new gate was run against the fault it exists to catch, with the sabotage asserted
present in the file
before the result was read, because a sabotage that silently no-ops
reads exactly like a pass:

Sabotage Result
**66 commits****14 commits** test_the_documents_commit_count_is_what_that_commit_carries FAILED
**12** merge commits**3** merge commits test_the_documents_merge_commit_count_is_what_that_commit_carries FAILED
rename the zizmor required context in main.json test_every_required_context_is_a_job_that_exists + …[zizmor] FAILED
delete the %G? trap paragraph test_the_document_does_not_read_local_signature_status_as_truth FAILED

Restored, all 12 pass.

What this does not do

The ruleset is still not applied, and this PR does not apply it. Re-confirmed
2026-09-06: repos/ChelseaKR/perimeter/rulesets is [] and main reports
"protected": false. Applying it is the owner's action — a POST adds a ruleset rather
than replacing one — and #15 stays open for it. bypass_actors is untouched and still
carries exactly the owner's RepositoryRole 5 / always entry.

Refs #15.

Prepared with AI assistance; reviewed before submission.

@ChelseaKR

Copy link
Copy Markdown
Owner Author

verify is red here, and not because of this branch. Every other check passes
(secret-scan, sast, zizmor, codeql). The five failures are all in
tests/test_release_claims.py, which this branch does not touch, and they all say the
same thing:

AssertionError: the tag list here cannot be trusted: this is a shallow clone.
actions/checkout fetches no tags at the default depth ... Check out with fetch-depth: 0

ci.yml does set fetch-depth: 0 on the verify job, and this branch does not modify
ci.yml (git diff origin/main origin/<head> -- .github/workflows/ci.yml is empty).

What was ruled out

Control Result
main's own CI, re-run just now (push event) passes
power-content-check PR #54, whose test_release_claims.py uses byte-identical shallow detection passes on a pull_request event, and has since merged
release/version-tag-honesty (#68's branch, whose tree contains this same test), re-run just now passes on a pull_request event
this run, re-run twice fails both times, so it is not transient

The only tree difference between #68's passing branch and main is the concurrency block
from #69. tests/test_release_claims.py is byte-identical between them.

What the failure actually is

The checkout is not truncated. In the same job, on the same checkout, the new
tests/test_ruleset_evidence.py runs git cat-file -t d4f533f and
git rev-list --count d4f533f and passes, reading a 66-commit ancestry. All five failures
are in test_release_claims.py; none is in the module that walks history.

So the checkout has full history and git rev-parse --is-shallow-repository returns
true at the same time. That predicate reports whether a .git/shallow marker exists, not
whether anything is actually truncated, and in this repository's pull_request-event
checkouts today those two have come apart.

That is this portfolio's dominant defect class pointed the other way: a state marker read
as a measurement
. The gate is failing closed, which is the correct direction and
exactly what it was built to do, but it is refusing to answer a question it can in fact
answer here.

What I did not do

I did not touch that gate. Weakening or re-baselining a version-honesty check that landed
an hour ago is not a call to make from inside an unrelated pull request, and "make it
green" is the wrong instinct for a check that is being conservative. Recording the
diagnosis instead, with the controls that produced it, so whoever fixes it starts from
evidence rather than from a red tick.

The fix is not a looser assertion: it is to make the predicate ask what the message
claims it asks, or to stop the PR-event checkout leaving a shallow marker behind. Both are
changes to test_release_claims.py or ci.yml, and both belong in their own pull request
where the negative control can be run against a genuinely truncated clone.

Prepared with AI assistance; reviewed before submission.

@ChelseaKR

Copy link
Copy Markdown
Owner Author

This PR's red verify job is not about this diff.

tests/test_release_claims.py refuses to read the tag list because git rev-parse --is-shallow-repository reports true. The message blames actions/checkout, but fetch-depth: 0 is set on the verify job and the logged fetch carries no --depth — the same refspec reproduced locally on git 2.55.0 produces a clone that is not shallow.

Measured on a runner via a throwaway diagnostic branch (now deleted): .git/shallow is absent after checkout, absent after make lock-check sync node-sync, absent after make browser-sync, and appears the moment tests/test_a11y_browser_gate.py runs — holding d4f533f, the pull request's base SHA. The cause is Playwright's git-info plugin: gitDiff opens with git fetch origin <pr base sha> --depth=1 ... (node_modules/playwright/lib/runner/index.js), and the harness's working directory is inside this work tree, so the fetch shallows the repository.

It presented as flake because it is a race: whether it fired depended on whether pytest -n auto scheduled the browser gate before the release-claims tests. Pushes to main were unaffected — with no PR base SHA there is nothing to fetch — so main stayed green while pull requests failed at random.

PR #74 turns the capture off (captureGitInfo: { commit: false, diff: false }) and strips the CI-identifying environment names the plugin reads, with negative controls for both halves. It is green.

Once #74 is on main, this PR needs no rebase: its merge ref is recomputed against the new base, so re-running the ci workflow here should be enough to get a real verdict.

Prepared with AI assistance; reviewed before submission.

@ChelseaKR ChelseaKR closed this Sep 7, 2026
@ChelseaKR ChelseaKR reopened this Sep 7, 2026
… has sixty-six

.github/rulesets/README.md recommends required_signatures and argues against merge
commits from measurements taken on 2026-08-15 and never re-read: "all fourteen commits
on main" and "the three merge commits already on main". At d4f533f there are 66 and 12.
Both conclusions survive; the evidence for them did not. The counts are now stated
against the commit they were taken at, so main advancing does not falsify them.

Re-measuring turned up a worse problem. A local git log --format='%G?' reports N, "no
signature", for thirteen commits on main. They are not unsigned: they are SSH-signed
with the owner's key, and git 2.55 classifies an SSH signature as N when no
gpg.ssh.allowedSignersFile is configured. Supply any allowed-signers file and the same
commit reports U. The other fifty-three are PGP-signed by GitHub's web-flow key and
report E. Zero commits on main carry no signature, all 66 report verification.verified:
true from the API, and that API verification is what GitHub's own required_signatures
rule enforces. An audit that read N as unsigned would report thirteen unsigned commits
that do not exist.

tests/test_ruleset_evidence.py holds both halves without becoming a counter that jams
the queue. The commit and merge counts are re-measured from the SHA the prose pins, so
history advancing changes nothing, while a number edited without moving the pin, a pin
moved without re-measuring, or a checkout too shallow to see that commit all fail. Every
required status check context in main.json is derived from the workflow files, so a
renamed job fails here rather than silently emptying the profile on the day it is
applied. Each gate was run against the fault it exists to catch, with the sabotage
asserted present in the file before the result was read.

The ruleset is still not applied; re-confirmed today that rulesets is [] and main
reports protected: false. Applying it is the owner's action and stays open as #15.
@ChelseaKR
ChelseaKR force-pushed the docs/ruleset-evidence-remeasured branch from dfeae2d to 73aade5 Compare September 7, 2026 06:21
@ChelseaKR
ChelseaKR merged commit a7e87c0 into main Sep 7, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant