You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(promexport): cg_extract_* has no component label, so an alert cannot say which component went underwater — and now inherits cost_source without expressing it #180
The cg_extract_* series on /metrics are process totals with no component label, so they cannot express the attribution /stats now carries — and after #178 they are actively misleading rather than merely coarse.
cg_extract_calls_total{outcome="made"} # summed across extract_llm AND extract_llm_sweep
cg_extract_cost_usd
cg_extract_net_value_usd
cg_extract_latency_ms
extract_llm and extract_llm_sweep both write the counters behind these. #176 measured what that costs a reader: 101 calls at 59,009 ms and a net value of −$1.162 attributed to extract_llm, whose own debug record showed zero surviving candidates on all 374 requests — the figures were the sweep's. #178 fixed /stats by adding extract.by_component; /metrics still publishes only the pooled number, so an alert that fires on cg_extract_net_value_usd still cannot say which component went underwater.
Why it got worse, not just left behind
#178 also added cost_source to /stats, because a spend figure of 0 and no evidence of spend are the same number and the opposite claim. The values are component (each call priced itself), host_total (the host's process-global cheap-model spend — a superset that also carries summarize and agentdiet), partial (some calls unpriced, so the total is a floor), unpriced, and none. The aggregate also names unpriced_components.
cg_extract_cost_usd and cg_extract_net_value_usd inherit whatever extraction_cost_usd computes, including the regime, and have no way to express which one they are in. A gauge reading −0.71 may be:
the components' own priced arithmetic (trustworthy), or
a floor that omits an unpriced component's dollars entirely (partial), or
the host's superset figure carrying two non-extraction components' spend (host_total).
Those support different decisions, and the series is identical in all three. An alert threshold tuned against one regime silently means something else in another. cg_extract_net_value_usd is documented in dash/metrics_export.go as the endpoint's only dollar figure, which is exactly why this matters.
Adding a component label changes the shape of series that off-repo, unversioned alert rules and dashboard queries are written against. sum(cg_extract_calls_total) keeps working; a rule matching the bare series name, or a graph grouping by the existing label set, does not. That is a monitoring break rather than a code change, and it wants its own rollout — which is why #178 confined the breakdown to /stats and stated this as a known gap.
What a fix needs to decide
Whether to label in place or add parallel series. Labelling cg_extract_* with component is the clean shape and the breaking one. Adding cg_extract_component_* alongside keeps the totals intact at the cost of two series families to keep consistent — and fix(metrics): scope extraction accounting per component, and log every extraction call #178's own experience is that two totals maintained in parallel is how the second drifts from the first (metrics/extract.go derives its aggregate by summing for exactly that reason).
How the cost regime becomes alertable. A number whose trustworthiness varies needs the regime beside it. Candidates: a cg_extract_cost_source info-style gauge labelled by source, or refusing to publish the cost gauges at all under partial/unpriced — fix(metrics): scope extraction accounting per component, and log every extraction call #178 already took the smaller version of that decision, having promexport omit cg_extract_net_value_usd rather than publish an unknown as 0, since 0 on that gauge reads as exactly break-even.
Cardinality. Bounded by code (the two extraction components), so this is not a label-explosion risk — worth stating because the gate-label series in promexport already assumes bounded-by-code cardinality.
Provenance
Raised in review of #178 and deferred there deliberately. /stats carries the per-component breakdown today; this is the monitoring surface catching up.
Symptom
The
cg_extract_*series on/metricsare process totals with nocomponentlabel, so they cannot express the attribution/statsnow carries — and after #178 they are actively misleading rather than merely coarse.extract_llmandextract_llm_sweepboth write the counters behind these. #176 measured what that costs a reader: 101 calls at 59,009 ms and a net value of −$1.162 attributed toextract_llm, whose own debug record showed zero surviving candidates on all 374 requests — the figures were the sweep's. #178 fixed/statsby addingextract.by_component;/metricsstill publishes only the pooled number, so an alert that fires oncg_extract_net_value_usdstill cannot say which component went underwater.Why it got worse, not just left behind
#178 also added
cost_sourceto/stats, because a spend figure of0and no evidence of spend are the same number and the opposite claim. The values arecomponent(each call priced itself),host_total(the host's process-global cheap-model spend — a superset that also carriessummarizeandagentdiet),partial(some calls unpriced, so the total is a floor),unpriced, andnone. The aggregate also namesunpriced_components.cg_extract_cost_usdandcg_extract_net_value_usdinherit whateverextraction_cost_usdcomputes, including the regime, and have no way to express which one they are in. A gauge reading−0.71may be:partial), orhost_total).Those support different decisions, and the series is identical in all three. An alert threshold tuned against one regime silently means something else in another.
cg_extract_net_value_usdis documented indash/metrics_export.goas the endpoint's only dollar figure, which is exactly why this matters.Why it was not done in #178
Adding a
componentlabel changes the shape of series that off-repo, unversioned alert rules and dashboard queries are written against.sum(cg_extract_calls_total)keeps working; a rule matching the bare series name, or a graph grouping by the existing label set, does not. That is a monitoring break rather than a code change, and it wants its own rollout — which is why #178 confined the breakdown to/statsand stated this as a known gap.What a fix needs to decide
cg_extract_*withcomponentis the clean shape and the breaking one. Addingcg_extract_component_*alongside keeps the totals intact at the cost of two series families to keep consistent — and fix(metrics): scope extraction accounting per component, and log every extraction call #178's own experience is that two totals maintained in parallel is how the second drifts from the first (metrics/extract.goderives its aggregate by summing for exactly that reason).cg_extract_cost_sourceinfo-style gauge labelled by source, or refusing to publish the cost gauges at all underpartial/unpriced— fix(metrics): scope extraction accounting per component, and log every extraction call #178 already took the smaller version of that decision, havingpromexportomitcg_extract_net_value_usdrather than publish an unknown as0, since0on that gauge reads as exactly break-even.promexportalready assumes bounded-by-code cardinality.Provenance
Raised in review of #178 and deferred there deliberately.
/statscarries the per-component breakdown today; this is the monitoring surface catching up.