refactor(#195): add silent-failure severity escalation guidance - #309
refactor(#195): add silent-failure severity escalation guidance#309fullsend-ai-coder[bot] wants to merge 1 commit into
Conversation
Add explicit severity escalation rules for silent-failure findings to
the correctness sub-agent, standalone code-review skill, and challenger
sub-agent. When code silently produces wrong results without any error
signal, severity must be medium or higher regardless of how unlikely the
trigger seems today.
Changes:
- correctness.md: new "Silent-failure severity escalation" section
setting a medium severity floor for silent failures and naming the
anti-pattern (probability-based severity discounting)
- challenger.md: carve-out preventing probability-based downgrading of
silent-failure findings
- code-review/SKILL.md: matching guidance in the standalone correctness
dimension
This addresses the systematic bias where the review agent rated
silent-failure bugs as [low] using probability reasoning ("X is likely
Y") rather than failure-mode reasoning ("silent and total when it
triggers"). Evidence: PR fullsend-ai#2947 annotated-tag finding rated [low] across
4+ passes while a human squad rated [HIGH].
Closes #195
E2E tests did not runE2E tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the See E2E testing guide for details. |
|
🤖 Finished Review · ✅ Success · Started 3:40 AM UTC · Completed 3:50 AM UTC |
ReviewFindingsMedium
Labels: PR modifies review skill documentation files exclusively. |
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
| (e.g., what happens if a critical sub-component fails — does the | ||
| caller degrade gracefully or silently proceed?). Trace the full path | ||
| from where the mechanism is set to where it is read. | ||
| - **Silent-failure severity escalation:** When a code path silently |
There was a problem hiding this comment.
[medium] internal-consistency
The SKILL.md silent-failure bullet covers "silently produces wrong results or skips critical work" while the authoritative correctness.md version covers "silently produces wrong results, returns stale or incorrect data, or skips critical work." The omission of "returns stale or incorrect data" narrows the SKILL.md rule: a standalone code review using SKILL.md could rate a stale-data-without-error-signal finding as low, while the same finding in a PR review (using correctness.md) would be escalated to medium or higher. Since stale data returned silently is precisely the class of bug this PR targets (the annotated-tag issue returned a stale/incorrect SHA with no error), the SKILL.md version should include the same qualifying condition.
Suggested fix: Add "returns stale or incorrect data" to the SKILL.md bullet so it reads: "When a code path silently produces wrong results, returns stale or incorrect data, or skips critical work without any error signal, rate the finding medium or higher..."
fullsend-ai-review
Bot
added
requires-manual-review
1 participant
Add explicit severity escalation rules for silent-failure findings to the correctness sub-agent, standalone code-review skill, and challenger sub-agent. When code silently produces wrong results without any error signal, severity must be medium or higher regardless of how unlikely the trigger seems today.
Changes:
setting a medium severity floor for silent failures and naming the
anti-pattern (probability-based severity discounting)
silent-failure findings
dimension
This addresses the systematic bias where the review agent rated silent-failure bugs as [low] using probability reasoning ("X is likely Y") rather than failure-mode reasoning ("silent and total when it triggers"). Evidence: PR fullsend-ai#2947 annotated-tag finding rated [low] across 4+ passes while a human squad rated [HIGH].
Closes #195
Post-script verification
agent/195-silent-failure-severity-escalation)a2afa8b179a351845733c680b839a39a82304db6..HEAD)