docs(#5266): add review autonomy evidence tracking document

[fullsend-ai-coder]

Summary

• Add docs/problems/review-autonomy-evidence.md as a structured evidence corpus tracking empirical observations from PRs where both agent and human review can be compared
• Document counter-evidence from PR feat(repos): add upgrade and upgrade-mint subcommands #4080 (semver/regex Go implementation, 2,379 additions, 11 files) where the human reviewer found all 5 medium+ severity issues that led to code changes
• Cross-reference the new document from autonomy-spectrum.md, code-review.md, and trustworthiness-evidence.md

Related Issue

Closes #5266

Changes

New file: docs/problems/review-autonomy-evidence.md

A problem document that serves as the autonomy-readiness evidence corpus. It:

• Tracks counter-evidence from PR feat(repos): add upgrade and upgrade-mint subcommands #4080 with a detailed findings table (5 human-only findings, 37.5% bot false positive rate, severity miscalibration)
• References related evidence issues (Counter-evidence for review autonomy: review agent missed 12 unique findings on large telemetry refactor PR #4510 #5251 counter-evidence, Track autonomy readiness: review agent on PR #4102 fully covered human review findings #4852/Track autonomy readiness: review agent fully covered human review on test-only sandbox PR #2986 #4532/Track autonomy readiness: review agent outperforms human review on mechanical consistency checks in konflux-ci/architecture PR #368 #4995 positive evidence)
• Identifies patterns: change types where agents underperform (regex, semver, cross-document consistency, operational security reasoning) vs. where they perform well (mechanical checks, convention adherence, security surface analysis)
• Discusses a potential change-type-based autonomy gate complementing the existing protected-path mechanism in post-review.sh
• Follows the existing problem document structure with open questions

Modified files

• docs/problems/autonomy-spectrum.md — added cross-reference to the evidence corpus in the graduation criteria section
• docs/problems/code-review.md — added cross-reference in the open questions section (review quality quantification)
• docs/problems/trustworthiness-evidence.md — added cross-reference in the Related section

Testing

• Verify markdown renders correctly
• Verify cross-references resolve to the correct document
• Verify website sidebar auto-discovers the new file (uses getMarkdownFiles() for problems section)

🤖 Generated with Claude Code

---

Closes #5266

Post-script verification

• Branch is not main/master (agent/5266-review-autonomy-evidence)
• Secret scan passed (gitleaks — 3e54f80755d4fe6d5a7abcc8c1f6ae5fc06aae58..HEAD)
• PR body secret scan passed (gitleaks — no-git)
• Pre-commit hooks passed (authoritative run on runner)
• Tests ran inside sandbox

[github-actions]

Site preview

Preview: https://bcba2620-site.fullsend-ai.workers.dev

Commit: a35111af2423f17a95e710b85824b945faf09c0a

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 7:58 AM UTC · Completed 8:13 AM UTC
Commit: a35111a · View workflow run →

[fullsend-ai-retro]

Retro: PR #5267 — docs(#5266): add review autonomy evidence tracking document

Timeline

Time (UTC)	Event	Run
Jul 17 19:42	Retro bot files issue #5266 (counter-evidence for review autonomy from PR #4080)	—
Jul 17 19:43	Triage dispatched → success (4 min)	29608561282
Jul 17 19:47	Code dispatched → success (10 min, created PR #5267)	29608841345
Jul 17 19:57	Review dispatched → failure (45 sec, token mint error)	29609448228
Jul 21 07:49	Human reviewer (rh-hemartin) approves, no comments	—
Jul 21 07:56	PR merged	—

Root cause: silent review agent failure

The review agent failed during token minting with: agent token minting failed: minted agent token contains unexpected characters for role review. The fullsend CLI (v0.31.0) fetched the harness from fullsend-ai/agents@def8cb52 but failed before starting a Claude session. No status comment was posted on the PR, so the human reviewer had no indication the review agent never ran.

This is a compound failure of three independent gaps:

1. No retry — the token mint error was likely transient (other review runs for the same role succeeded), but the CLI had no retry logic for this step
2. No PR notification — the failure occurred before the agent could post its "started" status comment, so the PR showed zero review agent activity
3. No auto re-dispatch — after the review failed, no mechanism retried the review automatically

The result: a 3.5-day gap between PR creation and merge, filled only by a human manually noticing and reviewing the docs-only change.

Existing issue coverage

All three improvement areas are already tracked by open issues. This retro provides additional supporting evidence:

• #5377 (retry with backoff for transient token minting failures) — this run is a concrete case where retry would have resolved the issue, since other review-role mints succeeded in adjacent runs
• #3635 / #3189 (post failure status comment when agent fails before creating its own status comment) — this run had zero PR-visible feedback after the review agent failed during harness initialization
• #1297 (dispatch-layer failures should notify the PR, not fail silently) — the "unexpected characters" mint failure is another variant of silent dispatch-layer failure
• #2711 (auto re-dispatch review after consecutive infrastructure failures) — would have recovered the review pipeline without human intervention
• #5382 (status comment should classify infrastructure failures and suggest retry) — relevant for cases where auto-retry is not yet available

Other observations

• Behaviour check: Showed "fail" on the PR but was actually a benign cancellation from two pull_request_target triggers racing in the same concurrency group (cancel-in-progress: true). Not actionable — docs-only PRs don't match behaviour-relevant file patterns anyway.
• Code agent quality: Excellent — 10-minute turnaround, clean single commit (121 additions, 4 files), well-structured evidence tracking document with proper cross-references to autonomy-spectrum.md, code-review.md, and trustworthiness-evidence.md.
• Token mint failure prevalence: Examined the last 50 review.yml runs (33 success, 9 failure, 8 cancelled). None of the other recent failures were token mint errors — they were post-review script failures. This suggests the "unexpected characters" error is rare/transient.

No new proposals

All identified improvements are covered by existing open issues. No new proposals are warranted.