What is missing
#188 added five cg_stash_* series to /metrics, and deploy/grafana/dashboards/context-guru.json references none of them:
cg_stash_reserve_entries{state="live"|"capacity"}
cg_stash_reserve_bytes{state="live"|"capacity"}
cg_stash_refused_total
cg_stash_missing_total
cg_stash_expired_total
So the reversibility budget is observable by scrape and invisible on the dashboard an operator actually looks at.
Why it matters more than a missing panel usually would
cg_stash_missing_total is the alertable one, and it is the only counter in the tree that says a reversibility promise has been broken: a <<cg:HASH>> marker went upstream with no payload behind it, so the agent can ask for content the proxy cannot produce. #187 was exactly that failure, 209 times in one benchmark arm, and it went unnoticed because nothing was watching.
cg_stash_refused_total is its opposite and must NOT be alerted on the same way — a refusal means the removal did not happen, content went upstream verbatim, and the cost is tokens rather than correctness. #184 added alerts "on the observability path"; this family arrived after it and got neither a row nor a rule.
How it was found
Auditing the #188 review round: the review asked for stash_missing to be split from stash_refused precisely because the two were indistinguishable, and the split is worth little if neither reaches a dashboard.
Suggested shape
- A reserve row with two panels: entries live/capacity, and bytes live/capacity. Both pairs together, because they are two budgets and which one binds decides which knob the operator turns (
max_entries vs stash_max_bytes).
rate(cg_stash_refused_total[5m]) as a graph — rising means the pipeline is declining work and saving less; a capacity signal, not an incident.- An alert on
increase(cg_stash_missing_total[15m]) > 0. Note it grows per turn per affected message rather than per distinct marker (a missing payload cannot be restored, so every later turn re-reports it), so alert on "non-zero", not on a rate threshold.
cg_stash_expired_total beside it, since missing and expired share a fix (ttl_seconds) that differs from refused's.
Docs for the fields are in docs/reference/routes.md under the rewind-reserve table.
What is missing
#188added fivecg_stash_*series to/metrics, anddeploy/grafana/dashboards/context-guru.jsonreferences none of them:So the reversibility budget is observable by scrape and invisible on the dashboard an operator actually looks at.
Why it matters more than a missing panel usually would
cg_stash_missing_totalis the alertable one, and it is the only counter in the tree that says a reversibility promise has been broken: a<<cg:HASH>>marker went upstream with no payload behind it, so the agent can ask for content the proxy cannot produce.#187was exactly that failure, 209 times in one benchmark arm, and it went unnoticed because nothing was watching.cg_stash_refused_totalis its opposite and must NOT be alerted on the same way — a refusal means the removal did not happen, content went upstream verbatim, and the cost is tokens rather than correctness.#184added alerts "on the observability path"; this family arrived after it and got neither a row nor a rule.How it was found
Auditing the #188 review round: the review asked for
stash_missingto be split fromstash_refusedprecisely because the two were indistinguishable, and the split is worth little if neither reaches a dashboard.Suggested shape
max_entriesvsstash_max_bytes).rate(cg_stash_refused_total[5m])as a graph — rising means the pipeline is declining work and saving less; a capacity signal, not an incident.increase(cg_stash_missing_total[15m]) > 0. Note it grows per turn per affected message rather than per distinct marker (a missing payload cannot be restored, so every later turn re-reports it), so alert on "non-zero", not on a rate threshold.cg_stash_expired_totalbeside it, sincemissingandexpiredshare a fix (ttl_seconds) that differs fromrefused's.Docs for the fields are in
docs/reference/routes.mdunder the rewind-reserve table.