Skip to content

feat(promexport): cg_extract_* has no component label, so an alert cannot say which component went underwater — and now inherits cost_source without expressing it #180

Description

@amiddavid

Symptom

The cg_extract_* series on /metrics are process totals with no component label, so they cannot express the attribution /stats now carries — and after #178 they are actively misleading rather than merely coarse.

cg_extract_calls_total{outcome="made"}   # summed across extract_llm AND extract_llm_sweep
cg_extract_cost_usd
cg_extract_net_value_usd
cg_extract_latency_ms

extract_llm and extract_llm_sweep both write the counters behind these. #176 measured what that costs a reader: 101 calls at 59,009 ms and a net value of −$1.162 attributed to extract_llm, whose own debug record showed zero surviving candidates on all 374 requests — the figures were the sweep's. #178 fixed /stats by adding extract.by_component; /metrics still publishes only the pooled number, so an alert that fires on cg_extract_net_value_usd still cannot say which component went underwater.

Why it got worse, not just left behind

#178 also added cost_source to /stats, because a spend figure of 0 and no evidence of spend are the same number and the opposite claim. The values are component (each call priced itself), host_total (the host's process-global cheap-model spend — a superset that also carries summarize and agentdiet), partial (some calls unpriced, so the total is a floor), unpriced, and none. The aggregate also names unpriced_components.

cg_extract_cost_usd and cg_extract_net_value_usd inherit whatever extraction_cost_usd computes, including the regime, and have no way to express which one they are in. A gauge reading −0.71 may be:

  • the components' own priced arithmetic (trustworthy), or
  • a floor that omits an unpriced component's dollars entirely (partial), or
  • the host's superset figure carrying two non-extraction components' spend (host_total).

Those support different decisions, and the series is identical in all three. An alert threshold tuned against one regime silently means something else in another. cg_extract_net_value_usd is documented in dash/metrics_export.go as the endpoint's only dollar figure, which is exactly why this matters.

Why it was not done in #178

Adding a component label changes the shape of series that off-repo, unversioned alert rules and dashboard queries are written against. sum(cg_extract_calls_total) keeps working; a rule matching the bare series name, or a graph grouping by the existing label set, does not. That is a monitoring break rather than a code change, and it wants its own rollout — which is why #178 confined the breakdown to /stats and stated this as a known gap.

What a fix needs to decide

  1. Whether to label in place or add parallel series. Labelling cg_extract_* with component is the clean shape and the breaking one. Adding cg_extract_component_* alongside keeps the totals intact at the cost of two series families to keep consistent — and fix(metrics): scope extraction accounting per component, and log every extraction call #178's own experience is that two totals maintained in parallel is how the second drifts from the first (metrics/extract.go derives its aggregate by summing for exactly that reason).
  2. How the cost regime becomes alertable. A number whose trustworthiness varies needs the regime beside it. Candidates: a cg_extract_cost_source info-style gauge labelled by source, or refusing to publish the cost gauges at all under partial/unpricedfix(metrics): scope extraction accounting per component, and log every extraction call #178 already took the smaller version of that decision, having promexport omit cg_extract_net_value_usd rather than publish an unknown as 0, since 0 on that gauge reads as exactly break-even.
  3. Cardinality. Bounded by code (the two extraction components), so this is not a label-explosion risk — worth stating because the gate-label series in promexport already assumes bounded-by-code cardinality.

Provenance

Raised in review of #178 and deferred there deliberately. /stats carries the per-component breakdown today; this is the monitoring surface catching up.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions