Expose supported Operator checker recovery

[Abiorh001]

The retained CheckerRetryService could safely supersede an exact finalized post-submit infrastructure failure, but Operators had no supported way to discover eligible sources or invoke that owner. This change adds project-scoped discovery and retry routes backed by fresh system Operator authority, signed bounded pagination, exact source/currentness checks, immutable receipt replay, concealed denials, and caller-root rollback/restaging.

The implementation extends the existing CHECKERS retry transaction. It binds the URL project during source selection before any foreign TASK lock, returns the same privacy-bounded source projection from discovery and retry, and keeps queued, running, business-completed, stale, and unresolved UNKNOWN effects out of retry eligibility. GET uses the canonical authorization-read rate control; POST uses the existing admin-mutation rate control and rejects non-human authority before CHECKERS or TASK lookup.

Prepare-time denials bind the original source/project, current identity link, and system scope before canonical denial evidence is retained. Internal retained-evidence, reservation, replay, and typed OUTBOX inconsistencies return bounded retryable 503 after rollback. Absent, foreign, stale, ineligible, and unauthorized sources stay concealed as 404; same-project request-key reuse with changed source or reason remains 409. Exact same-key replay freshly revalidates Operator authority, validates the retained original allow, and creates no second run, receipt, outbox event, initiation audit, or allow decision.

PR #589 is merged in main at 60e6302f54b614e9e285f7a9f4e8b32909caa415 and owns the shared operational-read audit migration, schema fingerprint, external-checker registry API, and full protected-route baseline. This branch is reconciled to that exact main ancestry, preserves the canonical migration and fingerprint byte-for-byte, and adds no sibling revision.

Validation for this reviewed candidate:

• focused real PostgreSQL/public API batch passed 4/4 in 143.68 seconds at product-source head 667e5a053: exact prepared denial; complete Operator discovery/retry/replay/revocation; real OUTBOX-conflict rollback; and auto-acceptance stale-source refusal. Isolated database, role, and MinIO cleanup all completed. The external-review repair at 0464524c8 then passed the complete public Operator journey 1/1 in 62.84 seconds, proving same-key replay and fresh-key superseded-source 404 with unchanged durable retry facts; owned cleanup completed.
• exact OpenAPI/action, full rate-control inventory and stray-route probe, closed owner, and lane catalogue/fingerprint selection passed 6/6 in 36.94 seconds after the final canonical prerequisite reconciliation.
• changed-file Ruff, behavior ownership, Commitrail, Markdown links, stale wording, git diff --check, and the workflow-exact test-structure validator pass.
• the denial regression remains AST-identical after moving into its existing authority lane to satisfy the structural bound; the moved test plus lane checks pass 3/3.
• architecture, security, QA, test-delta, product-operations, reuse, documentation, and focused structural-repair reviews passed; focused security/architecture and QA/test-delta/reuse replay passed on the stale-source repair at 0464524c8, and final evidence review passed at f71782657; main-relative freshness/docs replay passed at 5cbf4f3f4.

This PR is ready for human review. Merge eligibility remains blocked until the new exact-head required Backend aggregate passes and all review conversations are resolved. Source-specific uncertain-delivery reconciliation remains a later bounded change.

Related to #581.

Summary by CodeRabbit

• New Features
  • System Operators can discover eligible checker infrastructure failures for a specific project and retry them with an idempotency key.
  • Paginated results show limited source and retry-status information. Repeating the same retry request returns the existing result.
  • Retries are restricted to finalized, eligible failures; stale, unrelated, and uncertain-delivery cases remain unavailable.
• Documentation
  • Added guidance on the checker recovery workflow, permissions, and eligibility limits.

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 46 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: b23fd53c-9d14-4245-a20d-fe11d7b92c1e

📥 Commits

Reviewing files that changed from the base of the PR and between 60e6302 and 4a25843.

📒 Files selected for processing: 36

• .ci/behavior-ownership/partition.v1.json
• .commitrail/changes/mvp-operator-checker-recovery.md
• backend/app/adapters/auth/__init__.py
• backend/app/adapters/checkers/__init__.py
• backend/app/api/deps/checker_recovery.py
• backend/app/modules/authorization/catalogue.py
• backend/app/modules/authorization/checker_retry_authorization.py
• backend/app/modules/authorization/checker_retry_read.py
• backend/app/modules/authorization/domain/action_groups.py
• backend/app/modules/authorization/domain/audit_targets.py
• backend/app/modules/authorization/domain/checker_retry.py
• backend/app/modules/authorization/domain/checker_retry_reads.py
• backend/app/modules/authorization/domain/read_dispatch.py
• backend/app/modules/authorization/prepared.py
• backend/app/modules/authorization/runtime.py
• backend/app/modules/checkers/api/execution.py
• backend/app/modules/checkers/checker_retry.py
• backend/app/modules/checkers/execution_coordination.py
• backend/app/modules/checkers/router.py
• backend/scripts/behavior_ownership.py
• backend/scripts/test_lane_catalogue.py
• backend/tests/authorization/catalogue_fixtures.py
• backend/tests/authorization/setup_finalization/test_catalogue.py
• backend/tests/authorization/test_catalogue.py
• backend/tests/checkers/execution/test_checker_retry.py
• backend/tests/checkers/execution/test_checker_retry_authority.py
• backend/tests/checkers/execution/test_checker_retry_history_replay.py
• backend/tests/checkers/execution/test_checker_retry_http.py
• backend/tests/test_api_controls.py
• backend/tests/test_api_rate_controls.py
• backend/tests/test_behavior_ownership.py
• backend/tests/test_ci_lane_catalogue.py
• docs/operations_operator_workflow.md
• docs/operations_roles_permissions.md
• docs/roadmap_status.md
• mcp_server/contracts/authorization_context_get.json

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[durutheguru]

Review on behalf of the repository owner. Authorization, replay, cursor binding and rollback hold up, and this PR is the user of the operations.checker_retry_source.list audit pair that #589's migration admits. One fix before merge:

Should fix — a source that was already retried returns a retryable 503 instead of a concealed 404.
After Operator A retries source R, a second POST for R with a fresh Idempotency-Key passes the task lock, _source_from_run and authority prepare. _reserve_after_task_lock then raises CheckerExecutionUnavailable("checker_retry_source_unavailable") (execution_coordination.py:108), and checker_retry.py:394 maps every CheckerExecutionUnavailable to CheckerRetryOperationUnavailable → 503 checker_recovery_unavailable, retryable: true. No duplicate run is created, but clients will retry a permanently stale source forever, which contradicts the description ("stale … sources stay concealed as 404"). Suggested fix:

except CheckerExecutionUnavailable as exc:
    if str(exc) == "checker_retry_source_unavailable":
        raise CheckerRetrySourceUnavailable("checker_retry_source_unavailable") from exc
    raise CheckerRetryOperationUnavailable("checker_retry_reservation_unavailable") from exc

plus a regression test: retry R, then retry R again with a new key → 404, no new run, receipt or outbox event.

Optional.

• _append_retry_event: an OutboxIdempotencyConflict on checker-retry:<source_run_id> is deterministic but reported as retryable: true.
• discover() marks a source eligible from the fence alone; a source whose task has left evaluation_pending is listed as eligible and then refused with 404.
• discover(): a malformed row (UUID / pydantic ValidationError) surfaces as 500, while the retry path normalises the same failure to 503.

[durutheguru]

Approved on behalf of the repository owner. 0464524c8 fixes the superseded-source mapping: execution_coordination.py now raises CheckerRetrySourceUnavailable, checker_retry.py re-raises it as the concealed 404, and the authority test covers it. The optional notes (retryable OUTBOX key conflict, fence-only eligibility in discovery, 500 on malformed discovery rows) can be follow-ups. Merging once Backend CI against main passes.