Skip to content

refactor(#237): add silent-failure severity escalation guidance - #269

Closed
fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/237-silent-failure-severity-escalation
Closed

refactor(#237): add silent-failure severity escalation guidance#269
fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/237-silent-failure-severity-escalation

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown

Add explicit severity escalation rules for silent-failure findings to the correctness sub-agent, standalone code-review skill, and challenger sub-agent. When code silently produces wrong results without any error signal, severity must be medium or higher regardless of how unlikely the trigger seems today.

Changes:

  • correctness.md: new "Silent-failure severity escalation" section
    setting a medium severity floor for silent failures and naming
    the anti-pattern (probability-based severity discounting)
  • challenger.md: carve-out preventing probability-based downgrading
    of silent-failure findings
  • code-review/SKILL.md: matching guidance in the standalone
    correctness dimension

This addresses the systematic bias where the review agent rated silent-failure bugs as [low] using probability reasoning ("X is likely Y") rather than failure-mode reasoning ("silent and total when it triggers"). Evidence: PR fullsend-ai#2947 annotated-tag finding rated [low] across 4+ passes while a human squad rated [HIGH].


Closes #237

Post-script verification

  • Branch is not main/master (agent/237-silent-failure-severity-escalation)
  • Secret scan passed (gitleaks — c776c248ac9a91f10998ca7107e974a6c9d76514..HEAD)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

Add explicit severity escalation rules for silent-failure findings
to the correctness sub-agent, standalone code-review skill, and
challenger sub-agent. When code silently produces wrong results
without any error signal, severity must be medium or higher
regardless of how unlikely the trigger seems today.

Changes:
- correctness.md: new "Silent-failure severity escalation" section
  setting a medium severity floor for silent failures and naming
  the anti-pattern (probability-based severity discounting)
- challenger.md: carve-out preventing probability-based downgrading
  of silent-failure findings
- code-review/SKILL.md: matching guidance in the standalone
  correctness dimension

This addresses the systematic bias where the review agent rated
silent-failure bugs as [low] using probability reasoning ("X is
likely Y") rather than failure-mode reasoning ("silent and total
when it triggers"). Evidence: PR fullsend-ai#2947 annotated-tag finding rated
[low] across 4+ passes while a human squad rated [HIGH].

Closes #237
@github-actions

Copy link
Copy Markdown

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 10, 2026

Copy link
Copy Markdown

🤖 Finished Review · ❌ Failure · Started 12:58 PM UTC · Completed 12:58 PM UTC
Commit: c776c24 · View workflow run →

@guyoron1

Copy link
Copy Markdown
Owner

/fs-review

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 10, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 3:57 PM UTC · Completed 4:06 PM UTC
Commit: d8e3df7 · View workflow run →

@fullsend-ai-review

Copy link
Copy Markdown

Review — Approve

PR: #269 — refactor(#237): add silent-failure severity escalation guidance
Scope: 3 files changed (30+/1−) in internal/scaffold/fullsend-repo/skills/

Summary

This PR adds explicit severity escalation rules for silent-failure findings to three agent instruction files: the correctness sub-agent, the standalone code-review skill, and the challenger sub-agent. The change is well-motivated by documented evidence (issue #237) showing the review agent systematically rated silent-failure bugs as [low] using probability reasoning across 4+ review passes, while a human squad rated the same finding [HIGH] using failure-mode reasoning.

Dimensions reviewed

Dimension Result
Correctness ✅ No issues. New guidance is internally consistent across all three files. The medium or higher floor does not create a ceiling (permits critical). The challenger exception is well-scoped with a clear predicate ("no error signal"). No contradictions with severity anchoring rules.
Security ✅ No secrets, injection patterns, permission changes, or Unicode anomalies.
Intent & coherence ✅ Traces to issue #237. Scope is appropriate — adding guidance to the challenger and standalone code-review skill is necessary for completeness.
Style & conventions ✅ One initial finding about ### vs **bold:** heading format was removed after adversarial challenge — the ### heading is the correct structural choice for a policy/meta-guidance section.
Docs currency ✅ No documentation is made stale by this change.
Cross-repo contracts ⏭ Skipped — no exported interfaces or API surfaces modified.

Labels: PR modifies agent instruction/skill documentation files in the scaffold

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge documentation Improvements or additions to documentation labels Jul 10, 2026
@guyoron1

Copy link
Copy Markdown
Owner

/fs-fix

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 10, 2026

Copy link
Copy Markdown
Author

🤖 Finished Fix · ✅ Success · Started 5:38 PM UTC · Completed 5:43 PM UTC
Commit: d8e3df7 · View workflow run →

@fullsend-ai-coder

Copy link
Copy Markdown
Author

🔧 Fix agent — iteration 1 (human-triggered)

No fixes needed. The review bot approved PR #269 with zero findings across all dimensions. The human /fs-fix trigger provided no specific instruction. No code changes were made.

Disagreed (1):

  1. no actionable findings to fix: The review bot approved PR refactor(#237): add silent-failure severity escalation guidance #269 with no findings — all six review dimensions (correctness, security, intent & coherence, style & conventions, docs currency, cross-repo contracts) passed clean. The human trigger (/fs-fix) included no specific instruction (HUMAN_INSTRUCTION='none'). There are no code changes to make.

Tests: passed

Updated by fullsend fix agent

@guyoron1 guyoron1 closed this Jul 12, 2026
@guyoron1
guyoron1 deleted the agent/237-silent-failure-severity-escalation branch July 12, 2026 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ready-for-merge All reviewers approved — ready to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RTK+Ponytail] Evidence for #1144: annotated-tag silent-failure bug consistently rated [low] while human rated [high]

1 participant