Tested Prometheus alert rules for Kubernetes & node_exporter — with promtool unit tests and runbooks
12 production-ready Prometheus alerting rules for the failures that wake people up most: CrashLoopBackOff, OOMKilled containers, stuck Deployment rollouts, NotReady pods and nodes, full disks, and down hosts. Every alert ships with:
- promtool unit tests — one proving it fires on the failure pattern, one proving it stays quiet on healthy input
- a runbook — meaning, impact, diagnosis (
kubectl+ PromQL), mitigation, escalation - consistent labels and annotations —
severity,summary,description,runbook_url - tuned
for:durations — nofor: 0mnoise during rollouts and restarts
Works with Prometheus 2.42+ and 3.x, kube-prometheus-stack / prometheus-operator, Grafana Mimir, Thanos and VictoriaMetrics. Metric names match kube-state-metrics v2.x and node_exporter 1.x.
By Fractal Techware. MIT licensed.
| Alert | Severity | for: |
What it catches | Runbook |
|---|---|---|---|---|
KubePodCrashLooping |
warning | 15m | Container stuck in CrashLoopBackOff | runbook |
KubeContainerOOMKilled |
warning | – (keep_firing_for: 10m) |
Container restarted after exceeding its memory limit | runbook |
KubePodNotReady |
warning | 15m | Pod Pending / Unknown / Failed (Job pods excluded) | runbook |
KubeImagePullBackOff |
warning | 10m | ImagePullBackOff, ErrImagePull, InvalidImageName | runbook |
KubeDeploymentReplicasMismatch |
warning | 15m | Fewer available replicas than desired, no rollout in progress | runbook |
KubeDeploymentRolloutStuck |
warning | 15m | Rollout exceeded its progress deadline | runbook |
KubeNodeNotReady |
warning | 15m | Kubernetes node not Ready | runbook |
NodeExporterDown |
critical | 5m | Host or node_exporter unreachable | runbook |
NodeFilesystemSpaceFillingUp |
warning / critical | 1h | Disk predicted full within 24h / 4h (predict_linear) |
runbook |
NodeFilesystemAlmostOutOfSpace |
warning / critical | 30m | Less than 10% / 5% disk space left | runbook |
NodeHighCPUUsage |
warning | 30m | Host CPU above 90% | runbook |
NodeMemoryHighUtilization |
warning | 15m | Host memory utilisation above 90% | runbook |
12 alerts · 14 rules (two alerts have warning and critical variants) · 28 promtool unit tests · 12 runbooks.
git clone https://github.com/Fractal-Techware/prometheus-alert-rules.git
cd prometheus-alert-rules
./run-tests.sh # uses local promtool, or the prom/prometheus Docker image==> promtool check rules (3 files)
...
==> promtool test rules (3 files)
SUCCESS
All checks passed.
Copy rules/ next to your Prometheus config and add:
rule_files:
- /etc/prometheus/rules/*.ymlReload (curl -X POST http://localhost:9090/-/reload with --web.enable-lifecycle) and open Alerts.
Want to look around first? docker compose -f examples/docker-compose.yml up -d starts
Prometheus + node_exporter with the rules loaded at http://localhost:9090/alerts.
Wrap each rule file in a PrometheusRule object with yq v4 and apply:
for f in rules/*.rules.yml; do
N="$(basename "$f" .rules.yml)" yq '{"apiVersion": "monitoring.coreos.com/v1", "kind": "PrometheusRule",
"metadata": {"name": strenv(N), "namespace": "monitoring", "labels": {"release": "kube-prometheus-stack"}},
"spec": {"groups": .groups}}' "$f"
echo "---"
done | kubectl apply -f -The release label must match your prometheus.prometheusSpec.ruleSelector. kube-prometheus-stack's
default rules include alerts with the same names (e.g. KubePodCrashLooping); disable the
overlapping defaultRules.rules.kubernetesApps / nodeExporterAlerting groups to avoid duplicates.
mimirtool rules load rules/*.yml · Thanos Ruler --rule-file=rules/*.yml · vmalert -rule=rules/*.yml
runbook_url annotations are relative (runbooks/<domain>/<Alert>.md). Prefix them with
wherever you host the runbooks, for example:
sed -i.bak 's#runbook_url: runbooks/#runbook_url: https://github.com/<you>/<repo>/blob/main/runbooks/#' rules/*.yml tests/*.ymlMost alert rules are copied from a blog post or a snippet collection and never exercised until a real incident. Common failure modes:
- Metric renamed in kube-state-metrics v2 or node_exporter 1.x — the alert silently never fires.
- Label mismatch in a
on (...)join — the expression returns nothing. - No
for:— every rollout and restart pages someone. - Wrong exclusions — tmpfs, overlay or read-only filesystems trigger disk alerts.
A promtool unit test
feeds synthetic series into the rule and asserts the exact labels and annotations that fire at a
given time. Each alert here has at least one positive test and one negative test
(e.g. KubePodNotReady stays quiet for Job pods, NodeFilesystemAlmostOutOfSpace ignores tmpfs).
CI runs them on every push (.github/workflows/test.yml), and you can run them on your own
customised copy the same way.
rules/ alerting rules (3 groups: Kubernetes workloads, Kubernetes nodes, node_exporter)
tests/ promtool unit tests (*.test.yml)
runbooks/ one Markdown runbook per alert
examples/ prometheus.yml + docker-compose trial stack
run-tests.sh promtool check rules + test rules (local binary or Docker)
On call with an alert that isn't in this repo? https://fractal-techware.github.io/runbooks/ has a runbook page for all 165 alerts in the full pack: what the alert means, common causes and the first commands to run.
This repository is a free sample of the Prometheus Alert Rules & Runbook Pack, built from the same source and held to the same test standard. If these 12 alerts are useful, the paid tiers cover the rest of a production stack:
| Free (this repo) | Starter $19 | Pro $49 | Studio $99 | |
|---|---|---|---|---|
| Alerting rules | 12 alerts | 49 | 179 | 179 |
| Domains | K8s workloads/nodes, hosts | K8s workloads/nodes, hosts | 20 | 20 |
| promtool unit tests | 28 | 98 | 364 (+17 recording rules) | 364 |
| Runbooks | 12 | 46 | 165 | 165 |
| StatefulSets, DaemonSets, node pressure, capacity, inodes, network, RAID, clock | – | yes | yes | yes |
| Control plane, PVCs, Jobs, HPA, certificates, Prometheus/Alertmanager self-monitoring | – | – | yes | yes |
| PostgreSQL, Redis, Kafka, NGINX Ingress, Loki, CoreDNS, etcd, blackbox, SLO burn rates | – | – | yes | yes |
| PrometheusRule CRDs + tested Alertmanager routing | – | – | yes | yes |
| Generator (thresholds/labels from one file), Slack/PagerDuty/Teams/email templates, Helm values, HTML runbook site | – | – | – | yes |
| License | MIT | own organization | own organization | client / agency use |
See the full pack on Gumroad →
Issues and pull requests are welcome — see CONTRIBUTING.md. Every rule change needs a promtool test.
MIT © Fractal Techware