Problem
Every WasteFinding carries a single point estimate (estimated_savings.usd_per_session, tokens_per_session) and a tri-state WasteSeverity derived purely from USD thresholds. Consumers can't tell the difference between a finding we're highly confident about (e.g. five identical retry sequences across three sessions) and one that's plausible but thin (e.g. one bloated tool result, possibly a one-off). Likewise, savings is reported as a single number even when the realistic capture rate is a range — fixing a retry loop might recover 60–95% of its cost, but fixing bash output bloat by raising a config cap probably only recovers 20–50%.
Without confidence and capture-rate metadata, downstream surfaces (CLI tables, MCP, dashboards, agents) have to either trust every finding equally or invent their own filters.
Relevant files
crates/relayburn-sdk/src/analyze/findings.rs — WasteFinding, EstimatedSavings, WasteSeverity, severity_from_usd
crates/relayburn-sdk/src/analyze/patterns.rs — retry loops, replay storms, scroll churn, etc. emit findings
crates/relayburn-sdk/src/analyze/tool_output_bloat.rs — tool_output_bloat_to_finding, static-config vs. observed split
crates/relayburn-sdk/src/analyze/hotspots.rs — top-N cost rollups that could carry confidence on attribution
crates/relayburn-cli/src/commands/hotspots.rs and summary.rs — surface confidence in CLI output
Suggested scope
Additive on the data model side, behavior-preserving for current consumers:
- Add a
Confidence enum (Low | Medium | High) and per-finding evidence count (occurrences, distinct sessions, distinct turns) to WasteFinding. Serialize as optional fields so existing JSON consumers keep working.
- Extend
EstimatedSavings with optional usd_per_session_low / usd_per_session_high and capture_rate_low / capture_rate_high (0.0–1.0). The existing usd_per_session stays as the midpoint.
- Per detector, choose capture-rate bounds that reflect how realistic the fix is. Retry loops fixed end-to-end have a higher capture ceiling than bash output truncation. These bounds are detector-local constants, not user-tunable.
- Drive confidence from concrete signals: number of occurrences, variation across argshash, whether the pattern shows up in a single session or multiple. Don't introduce a general scoring framework — per-detector rules are fine.
- Update CLI table renderers in
relayburn-cli to show a compact confidence indicator and (optionally, behind a flag) the range. Default output should remain readable at current width.
Definition of done
WasteFinding carries optional confidence + occurrence-count + range fields; existing fields unchanged.
- At least three detectors (retry loops, tool output bloat, one cache/ghost-surface pattern) populate the new fields with non-default values.
- Sample CLI output for
burn hotspots / burn summary shows confidence without breaking the existing column layout.
- JSON output is backwards compatible (new fields are optional).
cargo test --workspace passes; tests assert that low-evidence findings get Low confidence and high-evidence ones get High.
Out of scope
- A general statistical confidence framework. Per-detector heuristics are sufficient.
- Changing
WasteSeverity semantics. It stays USD-threshold based.
- Recomputing historical findings on schema migration — new fields default to
None/midpoint behavior on old data.
Problem
Every
WasteFindingcarries a single point estimate (estimated_savings.usd_per_session,tokens_per_session) and a tri-stateWasteSeverityderived purely from USD thresholds. Consumers can't tell the difference between a finding we're highly confident about (e.g. five identical retry sequences across three sessions) and one that's plausible but thin (e.g. one bloated tool result, possibly a one-off). Likewise, savings is reported as a single number even when the realistic capture rate is a range — fixing a retry loop might recover 60–95% of its cost, but fixing bash output bloat by raising a config cap probably only recovers 20–50%.Without confidence and capture-rate metadata, downstream surfaces (CLI tables, MCP, dashboards, agents) have to either trust every finding equally or invent their own filters.
Relevant files
crates/relayburn-sdk/src/analyze/findings.rs—WasteFinding,EstimatedSavings,WasteSeverity,severity_from_usdcrates/relayburn-sdk/src/analyze/patterns.rs— retry loops, replay storms, scroll churn, etc. emit findingscrates/relayburn-sdk/src/analyze/tool_output_bloat.rs—tool_output_bloat_to_finding, static-config vs. observed splitcrates/relayburn-sdk/src/analyze/hotspots.rs— top-N cost rollups that could carry confidence on attributioncrates/relayburn-cli/src/commands/hotspots.rsandsummary.rs— surface confidence in CLI outputSuggested scope
Additive on the data model side, behavior-preserving for current consumers:
Confidenceenum (Low | Medium | High) and per-finding evidence count (occurrences, distinct sessions, distinct turns) toWasteFinding. Serialize as optional fields so existing JSON consumers keep working.EstimatedSavingswith optionalusd_per_session_low/usd_per_session_highandcapture_rate_low/capture_rate_high(0.0–1.0). The existingusd_per_sessionstays as the midpoint.relayburn-clito show a compact confidence indicator and (optionally, behind a flag) the range. Default output should remain readable at current width.Definition of done
WasteFindingcarries optional confidence + occurrence-count + range fields; existing fields unchanged.burn hotspots/burn summaryshows confidence without breaking the existing column layout.cargo test --workspacepasses; tests assert that low-evidence findings getLowconfidence and high-evidence ones getHigh.Out of scope
WasteSeveritysemantics. It stays USD-threshold based.None/midpoint behavior on old data.