Review agent should apply elevated severity when findings co-occur with breaking-change indicators

[fullsend-ai-retro]

What happened

On PR #4510, the review agent and human reviewer (waynesun09) independently identified 5 of the same issues. In all 5 cases, the bot rated the finding 1–2 severity levels lower than the human:

Finding	Bot severity	Human severity	Gap
OTEL_SDK_DISABLED doc mismatch	MEDIUM	MEDIUM	Same
parentSampledProcessor child-span leak	MEDIUM	HIGH	1 level
NaN/Inf float drops entire span	LOW	MEDIUM	1 level
Attribute rename agent → fullsend.agent	MEDIUM	HIGH	1 level
No fsync (crash-safety regression)	LOW	MEDIUM	1 level
Missing TRACEPARENT adoption tests	LOW	MEDIUM	1 level

The PR carried multiple breaking-change signals: ! suffix in the title, tech-debt label, ~1,540 lines of deleted test code, 6 deleted source files, and a complete subsystem replacement. Despite these strong signals of high behavioral-contract exposure, the bot applied standard severity calibration.

The human reviewer recognized that in the context of a breaking refactor, issues like "NaN/Inf float edge case" are not LOW-severity curiosities — they represent forensic data loss in the replacement system (dropping entire spans from run-telemetry.jsonl). The severity depends on the stakes of the change, not just the technical likelihood of the edge case.

What could go better

The review agent should incorporate contextual severity modifiers based on PR-level signals. When a PR exhibits breaking-change indicators (title contains !, deletes >500 lines of test code, replaces an entire subsystem, carries breaking-change labels), findings related to behavioral contracts, API surfaces, and data integrity should receive an automatic severity boost.

This is distinct from the deleted-test invariant analysis (proposal 1). That proposal addresses which findings to generate; this proposal addresses how to rate findings the agent already generates. The bot correctly identified the NaN/Inf edge case and the attribute rename — it just under-weighted their impact.

Confidence: MEDIUM-HIGH. The pattern is consistent across all 5 overlapping findings on this PR. However, I have not verified whether this under-rating pattern holds across other PRs — it may be specific to the scale and breaking nature of this particular change.

Risk: Over-correction could lead to severity inflation on routine refactors, increasing review noise. The modifier should be calibrated against the degree of breaking-change exposure, not applied as a blanket boost.

Proposed change

Add contextual severity modifiers to the review agent's finding-rating logic. Specifically:

1. Detect breaking-change exposure at the PR level before rating individual findings. Signals to check:

   • PR title contains ! (conventional commits breaking-change marker)
   • PR deletes test files or removes >30% of test coverage in touched packages
   • PR deletes source files that are replaced by new files (subsystem replacement pattern)
   • PR carries labels indicating high-impact change (e.g., tech-debt, breaking)

2. Apply severity modifiers when breaking-change exposure is detected:

   • Findings about behavioral contract changes (attribute renames, API surface changes, format changes): boost by 1 level (LOW→MEDIUM, MEDIUM→HIGH)
   • Findings about data integrity in replacement systems (edge cases that cause data loss, silent failures): boost by 1 level
   • Findings about test-adequacy gaps in replacement code: boost by 1 level (these are more consequential when old tests are deleted)

3. Annotate the severity boost in the finding description so reviewers understand why a finding was rated higher than usual (e.g., "Severity elevated from LOW to MEDIUM due to breaking-change context: PR deletes the test that previously guarded this invariant").

This is a change to the review skill or review agent's system prompt, likely at internal/scaffold/fullsend-repo/skills/ or the agent definition. The severity modifiers should be documented in the review agent's rating rubric.

Validation criteria

1. Alignment with human severity: On PRs with breaking-change indicators, the review agent's severity ratings for overlapping findings should match human reviewer ratings within 1 level in >80% of cases (currently 17% exact match, 83% under-rated by 1 level on this PR).
2. No severity inflation on non-breaking PRs: On routine PRs without breaking-change indicators, severity ratings should remain unchanged from current behavior.
3. Boost annotation present: When a severity modifier is applied, the finding description should include a brief explanation of why the severity was elevated.
4. Measurable on historical PRs: Re-run the review agent on PR refactor(telemetry)!: replace bespoke recorder with OTel Go SDK #4510's final commit with the severity modifiers enabled. At least 3 of the 5 overlapping findings should be rated at the same level as the human reviewer's rating.

---

Generated by retro agent from #4510

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 3:20 PM UTC · Completed 3:24 PM UTC
Commit: cc7a526 · View workflow run →

[fullsend-ai-triage]

Triage Summary

Severity: Medium · Category: Feature

Problem

The review agent's severity calibration is static and does not incorporate PR-level context. On breaking-change PRs, findings about behavioral contracts, data integrity, and test adequacy are consistently rated 1 level below human reviewer assessments. Evidence from PR #4510 shows 4 of 5 overlapping findings under-rated by 1 level.

Root Cause Hypothesis

The pr-review skill evaluates finding severity in isolation based on technical likelihood, without detecting PR-level breaking-change signals (title ! suffix, large test deletions, subsystem replacement patterns, breaking-change labels). These signals indicate elevated behavioral-contract exposure that should boost severity for relevant finding categories.

Recommended Fix

Add contextual severity modifiers to the pr-review skill definition (likely internal/scaffold/fullsend-repo/skills/pr-review/SKILL.md or meta-prompt.md):

1. Detect breaking-change exposure at the PR level before rating findings
2. Boost severity by 1 level for findings about behavioral contracts, data integrity in replacement systems, and test-adequacy gaps
3. Annotate each boost with a brief rationale in the finding description

Risk: over-correction on routine refactors. The modifier should be gated on the degree of breaking-change exposure, not applied as a blanket boost.

Related Work

• Review agent should analyze deleted test assertions to detect behavioral regressions in replacement code #5249 — Complementary proposal about which findings to generate (deleted test analysis); this issue is about how to rate findings the agent already generates
• Counter-evidence for review autonomy: review agent missed 12 unique findings on large telemetry refactor PR #4510 #5251 — Parent evidence tracking issue from the same PR analysis
• Review agent should not downgrade finding severity based on speculation about unseen diff content or assumed bot behavior #5128, Review agent should elevate mixed-scope PR detection from style/low to medium severity #5135 — Other severity calibration improvements (different mechanisms)

Proposed Test Case

Scenario: Severity elevation on breaking-change PR
  Given PR #4510 final commit with severity modifiers enabled
  When the review agent rates the 5 overlapping findings
  Then at least 3 findings match human severity ratings
  And each elevated finding includes a boost annotation

Scenario: No severity inflation on non-breaking PR
  Given a routine PR without breaking-change indicators
  When the review agent rates findings
  Then severity ratings are unchanged from current behavior

Edge cases:
  - PR with `!` in title but minimal deletions → should trigger
  - PR with large deletions but no `!` → should trigger on deletion signal
  - Routine refactor without breaking signals → should NOT trigger

---

Labels: This is a feature enhancement to the review agent's severity calibration logic, touching the skills component. Medium priority — impacts review quality on breaking-change PRs but not blocking.