Installing Prometheus and Grafana is table stakes. This lab demonstrates actually operating services reliably — SLO-based alerting, error budget burn-rate math, correlated debugging across services, and chaos engineering with documented expected outcomes.
Most observability demos stop at "install the stack." This lab goes further: define SLOs with real error budget math, build multi-window burn-rate alerts (not threshold alerts), write runbooks linked from alert annotations, inject faults and document exactly which dashboards change and which alerts fire. The observability stack exists to answer on-call questions, not to generate pretty graphs.
- SLO definitions with error budget math: 99.5% availability, 95% < 500ms latency, documented in docs/slo-definitions.md
- Multi-window burn-rate alerting: Per the Google SRE Workbook — not threshold alerts. Fast burn (14.4x, critical) and slow burn (6x, warning) with documented threshold derivation
- Alertmanager routing: Real routing tree with severity-based receivers, grouping by
[alertname, slo], and inhibition rules (critical suppresses warning for the same SLO) - Request ID correlation: UUID propagated API → Redis → Worker → all structured log lines, queryable in Loki
- 4 Grafana dashboards as code: SLO overview, API RED metrics, worker metrics, request tracing
- 4 runbooks linked from alerts: Each alert's
runbook_urlannotation points to an actionable document - 5 chaos scenarios with expected behavior: Shell scripts that inject faults, paired with docs describing what should happen on dashboards and which alerts fire
- CI for observability configs:
promtool test rulesvalidates alert math, dashboard JSON syntax checked, runbook links verified - Multi-service Go application: API + async worker + PostgreSQL + Redis — multiple failure domains to observe
A task queue system — deliberately simple so the observability patterns are the focus.
API Service (cmd/api/):
| Endpoint | Method | Description |
|---|---|---|
/api/v1/tasks |
POST | Submit a new task |
/api/v1/tasks/{id} |
GET | Get task status/result |
/api/v1/tasks |
GET | List recent tasks (with ?status= filter) |
/health |
GET | Liveness probe |
/ready |
GET | Readiness probe (DB + Redis) |
/metrics |
GET | Prometheus metrics |
Worker Service (cmd/worker/): Dequeues tasks from Redis, computes SHA-256 hash of payload, writes result to PostgreSQL. Exposes /metrics on port 9090.
| SLO | Target | Error Budget (30d) | Fast Burn Alert | Slow Burn Alert |
|---|---|---|---|---|
| Availability | 99.5% non-5xx | 0.5% (~3.6h outage) | 14.4x burn rate → critical | 6x burn rate → warning |
| Latency | 95% < 500ms | 5% slow requests | 14.4x burn rate → critical | 6x burn rate → warning |
Full derivation: docs/slo-definitions.md
./scripts/prerequisites.sh./scripts/setup-local.sh
kubectl -n sre-lab port-forward svc/api 8080:80# Submit a task
curl -X POST http://localhost:8080/api/v1/tasks \
-H "Content-Type: application/json" \
-d '{"payload":"hello world"}'
# Check task status
curl http://localhost:8080/api/v1/tasks/{id}
# List all tasks
curl http://localhost:8080/api/v1/tasks# Generate sustained load
./scripts/generate-load.sh &
# Inject a fault
./chaos/scenarios/redis-unavailable.sh
# Read what should happen
cat ./chaos/expected/redis-unavailable.md
# Resolve
kubectl -n sre-lab delete networkpolicy chaos-block-redissre-observability-lab/
├── cmd/
│ ├── api/ # API server entrypoint
│ └── worker/ # Worker entrypoint
├── internal/
│ ├── config/ # Env-based config for both services
│ ├── handler/ # HTTP handlers + tests
│ ├── middleware/ # Request ID, logging, Prometheus metrics
│ ├── model/ # Domain types (Task)
│ ├── store/ # PostgreSQL repository (pgx/v5)
│ ├── queue/ # Redis job queue (go-redis/v9)
│ ├── worker/ # Task processor with chaos knobs
│ └── metrics/ # Prometheus collectors
├── migrations/ # SQL migrations (golang-migrate)
├── docker/
│ ├── Dockerfile.api # Multi-stage → distroless
│ ├── Dockerfile.worker
│ └── Dockerfile.migrations
├── k8s/
│ ├── api/ # Deployment, Service, HPA
│ ├── worker/ # Deployment, HPA
│ ├── postgres/ # StatefulSet (local dev)
│ └── redis/ # Deployment
├── observability/
│ ├── prometheus/rules/
│ │ ├── slo-availability.yml # Availability recording rules
│ │ ├── slo-latency.yml # Latency recording rules
│ │ └── alerts.yml # Burn-rate alerts + worker alerts
│ ├── alertmanager/
│ │ └── config.yml # Routing, grouping, inhibition
│ └── grafana/dashboards/
│ ├── slo-overview.json # SLO gauges, budget, burn rate
│ ├── api-service.json # RED metrics
│ ├── worker-service.json # Processing rate, queue depth
│ └── request-tracing.json # Log correlation by request_id
├── chaos/
│ ├── scenarios/ # 5 fault injection scripts
│ └── expected/ # What should happen for each
├── runbooks/ # Linked from alert annotations
│ ├── high-error-rate.md
│ ├── latency-slo-breach.md
│ ├── worker-queue-backup.md
│ └── database-connection-pool.md
├── tests/promtool/ # Alert rule unit tests
├── scripts/ # Setup, teardown, load gen, validation
├── docs/
│ ├── architecture.md
│ ├── slo-definitions.md # SLO contracts + error budget math
│ └── alerting-philosophy.md # Why burn rate, not thresholds
├── argocd/ # GitOps Application manifest
└── Taskfile.yml
| Decision | Rationale |
|---|---|
| Burn-rate over threshold alerts | Threshold alerts don't account for budget consumption. A brief spike is noise; sustained degradation is an incident. Burn rate encodes this. |
| Two windows per alert | Short window detects the incident; long window confirms it's sustained. Prevents both false positives (spikes) and stale alerts. |
| Shell scripts over chaos frameworks | Each scenario is 5 lines of kubectl. No framework to install, no abstraction to learn. The value is in the expected-behavior documentation, not the injection mechanism. |
| Chaos knobs via env vars | FAILURE_RATE and SLOW_LATENCY on the worker deployment. No code changes needed — just kubectl set env. Hot-reloadable via rolling restart. |
| Request ID in queue message | Enables end-to-end tracing without distributed tracing infrastructure. A UUID in the log line is queryable in Loki today. |
| promtool test rules in CI | Alert rules are code. They should be tested like code. Synthetic time series prove the math works. |
- Go Deploy Lab — The deployment lifecycle patterns this lab builds on
- K8s Bootstrap Lab — The platform (Prometheus, Grafana, Loki) this lab's observability configs target
- Container Hardening Lab — Container security patterns applied to the Dockerfiles here