ci: report test impact in shadow mode

[Abiorh001]

Change

WS-CI-006-01 - Shadow-mode backend test-impact report

Goal

Measure a conservative lane-impact recommendation alongside the complete existing backend suite before considering any future selective gate.

Intent And Planning Context

• Intent: avoid making every small PR pay for unrelated test lanes, without weakening current proof.
• Commitrail record: .commitrail/initiatives/WS-CI-006/WS-CI-006-01.md and initiative overview.

What Changed

• Added a PR-only exact-target impact report and retained all nine existing lanes, preflight, API integration proof, and required aggregate.
• Added a narrow initial module-to-lane map; lane ownership is derived from the canonical test catalogue. Unknown paths, shared support, renamed unmapped sources, malformed inputs, or selector errors recommend every lane.
• Bound the report to exact Git target facts when available; unavailable evidence is marked and never presented as verified.
• Added selector, rename, reporting, Git-delta, and workflow-isolation regressions. The impact report is not a dependency of the required test aggregate or API end-to-end step.

Why It Changed

The current workflow runs all 7,911+ tests regardless of likely impact. Shadow evidence lets us evaluate safe relevance before a separate human-reviewed decision about any required-test selection. This PR does not reduce CI runtime or enable selective execution.

Design Chosen

Use the existing semantic lanes as the initial coarse selector, with conservative all-lanes fallback. Keep the report advisory and separate from required test execution.

Alternatives Rejected

• Removing a percentage of tests.
• Enabling selective required checks before representative shadow evidence.
• Adding an external CI selection service or historical test-result authority.

Scope Control

Files Outside Stated Scope

• No product behavior, test assertions/collection, coverage policy, lane commands, or required checks changed.

Product Behavior

• No Workstream product behavior changed.

Evidence

Commands Run

backend/.venv/bin/python -m pytest -q backend/tests/test_ci_impact_selection.py backend/tests/test_ci_lane_catalogue.py backend/tests/test_ci_test_lanes.py scripts/test_lightweight_agent_gates.py scripts/test_git_delta.py scripts/test_commitrail_contracts.py scripts/test_commitrail_contribution_paths.py
backend/.venv/bin/python -m ruff check backend/scripts/test_impact_selection.py backend/tests/test_ci_impact_selection.py scripts/git_delta.py scripts/test_git_delta.py scripts/test_lightweight_agent_gates.py
WORKSTREAM_BASE_SHA=$(git merge-base origin/main HEAD) backend/.venv/bin/python scripts/check_commitrail_records.py --base-ref origin/main
backend/.venv/bin/python scripts/check_markdown_links.py
backend/.venv/bin/python scripts/check_stale_workstream_wording.py

Result Summary

120 passed, 97 subtests passed
Ruff, Commitrail, Markdown links, stale wording, workflow YAML parse, and git diff checks passed.
Hosted Backend CI is pending on this PR; all nine lanes remain required.

Acceptance Criteria Proof

• Shadow recommendation stays outside all required test dependencies and conditions.
• Unknown impact and classifier failure recommend all lanes.
• Full-suite local and workflow protection regressions pass.
• Exact-head hosted Backend workflow passes; pending.
• Observe reports alongside full results on representative future PRs before proposing any gate change.

Test Delta

Tests Added

• Selector target binding, impact closure, fallback, changed test, malformed/unsafe/duplicate input, markdown safety, unavailable evidence, and rename regressions.
• Workflow report-isolation and shared Git byte-output regressions.

Tests Modified

• Lightweight workflow/fan-in assertions now verify the report is independent of existing blocking execution.

Tests Removed Or Skipped

• None.

Impact-Routed Reviewer Results

Reviewed code SHA: 75ef4c4871c9c0200bada2c4932737f748b42705

Reviewer	Result	Blocking findings	Proof boundary and uncertainty
CI integrity	PASS	None	Source-level graph verified; hosted run pending.
QA	PASS	None	Includes real-Git rename and conservative fallback probes.
Test delta	PASS	None	Workflow coupling/API skip mutations detected.
Security	PASS	None	Advisory-only, read-only permissions, markdown injection checks.
Documentation	PASS	None	Record, runbook, roadmap and scope reconciled.
Reuse/dedup	PASS	None	Canonical lane catalogue and shared Git primitives reused.

External Review

CodeRabbit review has not yet run on this PR. Current GitHub check state should be checked after PR creation.

CI And Gate Integrity

• No workflow weakening.
• No lint/test/docstring gate weakening.
• Tests protect meaningful behavior; no tests removed or skipped.
• No package script weakening.
• No unpinned new GitHub Action.
• Checkout credential persistence disabled where checkout is used.

Remaining Risks

The initial impact map is deliberately incomplete, so most source changes recommend all lanes. Shadow output is candidate-produced and cannot validate changes to its own selector, map, catalogue, or workflow. It is not authority for later selective execution.

Follow-Up Work

Collect same-head shadow reports and full-suite outcomes from representative PRs. Any reduction to required execution needs a separate reviewed change and trusted selection design.

Human Review Focus

• Confirm the advisory report cannot suppress any existing suite or integration proof.
• Confirm unknown and rename cases conservatively select all lanes.
• Confirm no selective gating is introduced by this PR.

Human Merge Ownership

• I can explain what changed.
• I can explain why it changed.
• I know what could break.
• I accept the remaining risks.
• The user explicitly approved this specific PR for merge.

Summary by CodeRabbit

• New Features

  • Pull requests now include a downloadable test-impact report recommending backend test lanes and explaining the recommendations. When changes cannot be classified reliably, the report recommends all nine lanes.
  • The report is advisory only: all nine backend test lanes and existing checks remain required, and report failures do not affect them.

• Documentation

  • Updated backend testing and roadmap guidance to explain the report, its fallback behavior, and how recommendations relate to required test runs.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 724743e8-59a1-4838-8358-173c323a6353

📥 Commits

Reviewing files that changed from the base of the PR and between 539ea18 and dc2fa18.

📒 Files selected for processing (2)

• scripts/test_git_delta.py
• scripts/test_lightweight_agent_gates.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The change adds a backend test-impact selector and a pull-request-only report job. The report recommends test lanes from changed-path evidence, with all lanes as the fallback. Required lanes and existing checks continue to run independently.

Changes

Backend test-impact reporting

Layer / File(s)	Summary
Define mappings and classify changed paths .commitrail/initiatives/WS-CI-006/*, .commitrail/INDEX.md, .ci/test-impact/impact_map.json, scripts/backend_test_impact.py, scripts/git_delta.py, scripts/test_backend_test_impact.py, scripts/test_git_delta.py	Records the shadow-report scope and initial path mappings. The selector validates changed paths, resolves lane ownership, and falls back to all lanes when paths are unknown. Git output preserves NUL-delimited path bytes.
Bind and render impact reports scripts/backend_test_impact.py, scripts/test_backend_test_impact.py	Builds JSON and Markdown reports with target evidence, classification details, and digests. Report-generation failures produce an all-lanes fallback. Tests cover evidence, fallback behavior, path validation, and Markdown escaping.
Publish reports without changing required checks .github/workflows/backend.yml, .github/workflows/agent-gates.yml, scripts/test_backend_test_impact.py, scripts/test_lightweight_agent_gates.py, docs/operations_backend_testing.md, docs/roadmap_status.md	Adds a pull-request-only report job that uploads its outputs and links the artifact in the workflow summary. Tests and documentation specify that report recommendations do not control the required lane or aggregate execution.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Backend workflow
  participant backend_test_impact.py
  participant Artifact storage
  participant Workflow run summary
  Backend workflow->>backend_test_impact.py: Pass pull-request base and head SHAs
  backend_test_impact.py-->>Backend workflow: Write JSON and Markdown reports
  Backend workflow->>Artifact storage: Upload report files
  Backend workflow->>Workflow run summary: Add artifact URL

Loading

Merge Risk: ⚪ Minimal · up to dc2fa

This change adds an advisory test-impact report, and the required test lanes and checks are unchanged. No actionable merge-blocking risk was found in the supplied changes.

Security Architecture Review

Security architecture risk: 🔵 Low · up to dc2fa

The new recommendations remain advisory. Required test lanes and API checks do not consume them, and the reporting job has read-only repository permissions without persisted checkout credentials. No security regression was established, but validation covered source contracts rather than live cancellation, rerun, or publication behavior.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• observed — The added execution is a hosted CI reporting job with contents-read permission and checkout credential persistence disabled. Its configured outputs are report files, a CI upload and run-summary content; the job does not declare production secrets or deployment steps.

Security Findings and Attack Paths

• inferred — A pull-request author can influence selector code, mapping and therefore recommendation content published by CI. That influence is not an established required-test bypass: neither the lane matrix nor the aggregate consumes report output. Exact-target checks identify the evaluated inputs but do not independently approve candidate-controlled policy.

Trust Boundaries and Controls

• observed — Pull-request target SHAs enter the script through environment variables rather than shell command interpolation. The byte-preserving Git helper executes argument lists without a shell, retains NUL-delimited output, and raises explicit errors for execution failure, timeout or nonzero exit.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 13.21% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 53 functions across 5 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise, specific, and accurately describes the primary change: adding a shadow-mode CI test-impact report.
Description check	✅ Passed	The description is detailed and covers the change, intent, design, scope, evidence, acceptance criteria, test delta, review results, risks, and follow-up work. It omits the optional Allowed Files Chan…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}