Skip to content

Triage bot: improve RCA with upstream evidence gathering #1065

@sentry-junior

Description

@sentry-junior

The triage bot's RCA comments tend to restate the reporter's own diagnosis rather than gathering independent upstream evidence. For schema or API contract issues, the bot should verify the actual mismatch by querying Sentry directly, rather than taking the reporter's analysis at face value.

Current behavior

  • Bot confirms the reporter's diagnosis without independent verification (analyze_issue_with_seer fails with ZodError: Failed to validate keys: autofix.status #1064 is a clear example: the bot echoed the ZodError/enum analysis the reporter already provided, added no new evidence)
  • For Sentry API issues, the bot doesn't query upstream to observe the actual response shape or confirm when the contract changed
  • Evidence trail is opaque — comments don't distinguish what the bot checked vs. what it inferred from the issue body, making it hard for maintainers to trust or act on

Gap

  • For schema/contract mismatches, the highest-value analysis would: call the relevant Sentry API endpoint and capture the actual response shape, cross-reference the MCP schema to confirm the exact field(s) that diverge, and surface that as grounded evidence with explicit "I called X and saw Y" framing
  • Without that, the bot comment is redundant with the issue body and doesn't accelerate resolution

Suggested improvements

  • When an issue references a Sentry API response or schema field, have the bot make the relevant API call and include the observed response shape in its analysis
  • Require the bot to explicitly label what it verified vs. what it inferred (e.g., "Confirmed by calling GET /organizations/{org}/issues/{id}/autofix/ — response shows status: completed")
  • For schema mismatch issues, cross-reference Sentry source or API docs to confirm the current contract and when it diverged from the local expectation

Reference: #1064 (bot comment at #1064 (comment) restates the reporter's diagnosis without new evidence)

Action taken on behalf of immutable dcramer.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions