Count-drift gates run per-PR and cannot see cross-PR drift

[MSKazemi]

Merging #133 and #138 turned main red, and neither PR was at fault.

Both were green on their own head SHA. Both were branched 31 commits back, and both added tests. tests/test_doc_claims.py compares a hard-coded server-suite count in AGENTS.md against a live collect — neither PR's CI run ever saw the other's tests, so each was individually consistent and the merge of the two was not. 1040 claimed, 1047 collected. Tests (server) and Tests (server · py3.13) are both required checks, so main went red and stayed blocking until #152.

This is the same count-drift family as the earlier playbook-count problem, but with a new trigger: it is not reachable from any single PR. A per-PR gate structurally cannot see a number that only two merges together move, so no amount of care on the contributor side prevents it.

Options

1. Run the doc-claims gate on main after merge — catches it, but only after main is already red. Cheap, and at least the signal is immediate and unambiguous.
2. Have the PR job collect against the merge result rather than the PR head. GitHub already builds a merge commit for pull_request events; collecting there would have caught this before either merge. Better, and probably the real fix.
3. Stop hard-coding the count. The number in AGENTS.md is documentation of a fact the test suite already knows. A generated line, or a tolerance, removes the whole class — at the cost of losing a genuine tripwire that has caught real drift before.

Option 2 plus keeping the exact count is my current preference, but this is worth deciding rather than patching again.

Also worth noting

The same structural blind spot applies to every other hard-coded count the doc-claims checker guards (playbooks, detectors, flags, providers). None of them are safe from a two-PR interaction, only from a one-PR one.

[MSKazemi]

It happened again on 2026-09-09 — and this time the root cause is narrower than option 2 assumes, which makes the fix a settings change rather than a workflow change.

Fresh reproduction (a near-miss, not another red main)

Two PRs in flight yesterday, both adding tests, both branched from the same main:

PR	what check_doc_claims.py computed in isolation
#184 (parallel tool isolation, +5 tests)	5510
#187 (DNS playbook, +5 tests)	5510
actual total once both merged	5515

Exactly the shape described here: each PR was individually consistent, neither CI run could see the other's tests, and the merge of the two was not consistent.

main did not go red this time, and the reason is not reassuring: I happened to rebase #187 onto the merged main before merging it, which forced a recollection and surfaced 5515. Had I merged both without rebasing, main would have gone red exactly as it did after #133/#138. The current safeguard is a maintainer remembering to rebase — which is precisely the "no amount of care prevents it" property this issue is about, just relocated from the contributor to the maintainer.

Option 2 is already half-implemented — the missing piece is one setting

The issue proposes "have the PR job collect against the merge result rather than the PR head". That already happens. ci.yml uses actions/checkout with no ref: override, and for pull_request events the default checkout is the merge commit (refs/pull/N/merge), not the head. So the collection is already against PR + base.

The defect is that it is computed once, when the event fires, and never recomputed when the base moves:

$ gh api repos/MSKazemi/kubeintellect/branches/main/protection/required_status_checks --jq '{strict:.strict}'
{"strict":false}

strict is GitHub's "Require branches to be up to date before merging". With it off, merging PR A does not invalidate PR B's existing green checks and does not re-run them against the new base — so B merges on a merge-commit result computed against a main that no longer exists.

So the real statement of the bug is not "the PR job collects the wrong thing". It is "the PR job collects the right thing, at a moment that stops being true."

What that changes about the options

• Option 2 → set strict: true. No workflow edit at all; it forces a re-run against the current base before a merge is allowed, which recomputes the count exactly when it needs recomputing. This is the smallest change that closes the class.
• Honest cost, because it is not free: every PR must be up to date before merging, so concurrent PRs serialise and each needs a rebase-and-rerun. With Dependabot's batches that is real friction — I merged six Dependabot PRs yesterday, and under strict each would have re-run after the previous merge. Worth it or not is a judgement about PR volume, not about correctness.
• Option 1 (run the gate on main after merge) remains a good cheap complement even with strict: true, because it catches anything merged with --admin.
• Option 3 (stop hard-coding) still loses a tripwire that has caught real drift, most recently yesterday.

The blind spot is wider than this issue records

The closing note says the same applies to every hard-coded count the doc-claims checker guards. Since yesterday it applies to two more surfaces:

• CONTRIBUTING.md — its test counts were ungated and had rotted to 5× stale (~1,050 against 5515). Now on the same collection (docs(contributing): correct the test counts, and put them under the gate that already existed #191).
• A source comment. detectors/models.py said "The 21 promql: queries in the shipped playbooks" against a tree of 19, wrong for over two weeks, inside the comment that justifies refusing a promql-only detector. Now gated (fix(detectors): correct the unevaluated-promql count, and gate it so it cannot rot again #194).

Both were single-PR drift rather than cross-PR, so strict: true would not have caught either — they needed the gate's coverage widened, not its timing fixed. Worth separating those two failure modes when deciding here: timing is what this issue is about, and strict fixes it; coverage is a different axis and is being closed file by file.

⚠️ Branch protection is a repo-settings change, so I have not made it — recording the analysis for a decision.