fix(observability): SLO coherence -- rules, scrapes, retention, dashboard - #147
Merged
Conversation
…oard
Repo review round, observability findings:
- dashboard burn panel plotted slo:latency:error_budget7d (~0.05 when
healthy) on a positive-means-violation scale: permanent false red.
New slo:latency:error_budget_burn7d rule (budget - 0.05 allowance),
panel repointed; latency target rule added for symmetry.
- slo:error_rate:ratio7d and error_budget7d were byte-identical to the
availability rules and unused by the dashboard -- dropped, with a
NOTE in the rules file; burn now derives from actual_5xx_ratio7d.
- endpoint regex widened to include /items/{item_id}: the backend now
labels by route template (PR #146), making DELETE visible to SLOs
for the first time.
- Prometheus scrapes loki and alloy own metrics (previously only their
containers via cadvisor).
- Loki: deprecated table_manager block dropped (compactor handles
retention for tsdb); reject_old_samples_max_age aligned to the 24h
retention -- accepting week-old samples that get deleted immediately
made no sense.
- dashboard: 5xx stat max cap removed (incidents above 1% looked
pinned at the gauge edge); log panel filters container_name, the
label Alloy actually sets, instead of the Loki-synthesized
service_name.
- SLO.md: formulas synced to the shipped rules (error budget is the
consumed failure fraction, 1 - ratio), metric name corrected from
http_request_duration_seconds to http_request_latency_seconds --
copy-pasted queries now return data; grafana.ini comment matched to
its value.
Validated: promtool check config + rules (14 rules), loki
-verify-config, dashboard JSON parses.
3 tasks
vovinacci
added a commit
that referenced
this pull request
Jul 13, 2026
The subject-case rule read 'fix(observability): SLO coherence ...' as sentence-case and failed the PR title gate (#147). Subjects here start with acronyms routinely (SLO, API, CI, ADR); the rule is disabled -- type, scope, and the remaining conventional-commit structure stay enforced. Verified: the rejected title passes, garbage still fails.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Repo review round: the latency error-budget panel showed a permanent false violation, SLO.md documented a metric that does not exist, and several rule/config incoherences. Companion to #146 (which makes DELETE visible to the endpoint label these rules filter on).
Change model
prometheus_slo_rules.yml: newslo:latency:error_budget_burn7d+ latency target rule; duplicate error-rate ratio/budget rules dropped (byte-identical to availability, unused by the dashboard); endpoint regex widened with/items/{item_id}.prometheus.yml: loki + alloy own-metrics scrape jobs.loki/config.yml: deprecated table_manager removed (compactor owns retention on tsdb);reject_old_samples_max_agealigned to the 24h retention.container_name(the label Alloy sets).SLO.md: formulas and metric names synced to the shipped rules;grafana.inicomment matched to its value.Checklist
make cigreen locally; promtool check config + rules (14 rules), loki -verify-config, dashboard JSON validated🤖 Generated with Claude Code