Skip to content

Add confidence and capture-rate ranges to waste findings #460

Description

@willwashburn

Problem

Every WasteFinding carries a single point estimate (estimated_savings.usd_per_session, tokens_per_session) and a tri-state WasteSeverity derived purely from USD thresholds. Consumers can't tell the difference between a finding we're highly confident about (e.g. five identical retry sequences across three sessions) and one that's plausible but thin (e.g. one bloated tool result, possibly a one-off). Likewise, savings is reported as a single number even when the realistic capture rate is a range — fixing a retry loop might recover 60–95% of its cost, but fixing bash output bloat by raising a config cap probably only recovers 20–50%.

Without confidence and capture-rate metadata, downstream surfaces (CLI tables, MCP, dashboards, agents) have to either trust every finding equally or invent their own filters.

Relevant files

  • crates/relayburn-sdk/src/analyze/findings.rsWasteFinding, EstimatedSavings, WasteSeverity, severity_from_usd
  • crates/relayburn-sdk/src/analyze/patterns.rs — retry loops, replay storms, scroll churn, etc. emit findings
  • crates/relayburn-sdk/src/analyze/tool_output_bloat.rstool_output_bloat_to_finding, static-config vs. observed split
  • crates/relayburn-sdk/src/analyze/hotspots.rs — top-N cost rollups that could carry confidence on attribution
  • crates/relayburn-cli/src/commands/hotspots.rs and summary.rs — surface confidence in CLI output

Suggested scope

Additive on the data model side, behavior-preserving for current consumers:

  • Add a Confidence enum (Low | Medium | High) and per-finding evidence count (occurrences, distinct sessions, distinct turns) to WasteFinding. Serialize as optional fields so existing JSON consumers keep working.
  • Extend EstimatedSavings with optional usd_per_session_low / usd_per_session_high and capture_rate_low / capture_rate_high (0.0–1.0). The existing usd_per_session stays as the midpoint.
  • Per detector, choose capture-rate bounds that reflect how realistic the fix is. Retry loops fixed end-to-end have a higher capture ceiling than bash output truncation. These bounds are detector-local constants, not user-tunable.
  • Drive confidence from concrete signals: number of occurrences, variation across argshash, whether the pattern shows up in a single session or multiple. Don't introduce a general scoring framework — per-detector rules are fine.
  • Update CLI table renderers in relayburn-cli to show a compact confidence indicator and (optionally, behind a flag) the range. Default output should remain readable at current width.

Definition of done

  • WasteFinding carries optional confidence + occurrence-count + range fields; existing fields unchanged.
  • At least three detectors (retry loops, tool output bloat, one cache/ghost-surface pattern) populate the new fields with non-default values.
  • Sample CLI output for burn hotspots / burn summary shows confidence without breaking the existing column layout.
  • JSON output is backwards compatible (new fields are optional).
  • cargo test --workspace passes; tests assert that low-evidence findings get Low confidence and high-evidence ones get High.

Out of scope

  • A general statistical confidence framework. Per-detector heuristics are sufficient.
  • Changing WasteSeverity semantics. It stays USD-threshold based.
  • Recomputing historical findings on schema migration — new fields default to None/midpoint behavior on old data.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions