Skip to content

Context-size / cache-efficiency is not a first-class metric #505

Description

@willwashburn

Problem

Burn has no metric for how much context is being paid for per unit of output. The number that actually defined a recent spend incident — a 382:1 context-to-output token ratio — isn't surfaced anywhere in summary, hotspots, or findings. Nothing would have flagged that session as anomalous on cache/context-efficiency grounds even though it was the clearest signal available.

Evidence (2026-07-30 fleet spend forensics)

  • The incident session's defining characteristic was a 382:1 ratio of context tokens read to output tokens produced — i.e. enormous prompt re-processing for very little generated output.
  • Existing findings/hotspots rank by dollar cost (see Pricing table missing current-generation models — unpriced turns bury real hotspots #503) or raw token totals, neither of which isolates this shape of waste. A session can be cheap in absolute tokens and still be pathologically inefficient in context-to-output ratio, or vice versa.

Proposal

  1. Add context-tokens-per-output-token as a headline/summary metric, computed per turn and rolled up per session.
  2. Surface a per-session context-size distribution (e.g. p50/p95/max of context tokens per inference) so outlier sessions are visible without hand-computing the ratio.
  3. Consider a findings rule that flags sessions above a configurable context:output ratio threshold, independent of dollar cost.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions