diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bd044fd..f1358e5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -36,7 +36,7 @@ make chart-lint # helm lint + kubeconform [detector/src/promanomaly/detectors/](detector/src/promanomaly/detectors/), `pyproject.toml` entry points, and [docs/detectors.md](docs/detectors.md). CI checks all three. -- Adding a Helm value? Regenerate `values.schema.json` (`make chart-schema`). +- Adding a Helm value? Update [charts/promanomaly/values.schema.json](charts/promanomaly/values.schema.json) by hand to match. The schema is hand-authored alongside `values.yaml` so it can express constraints (enums, ranges, opt-in defaults) a generator can't infer from a sample — see the comment block above the `chart-lint` target in the [Makefile](Makefile). - Adding or changing a contracted label? Coordinate a PR pair against [LABELS_CONTRACT.md](LABELS_CONTRACT.md) here and in promforecast — CI diffs the two files and will fail if they drift. diff --git a/README.md b/README.md index d66f9f7..e91e78c 100644 --- a/README.md +++ b/README.md @@ -62,11 +62,18 @@ operational guidance — including [`docs/patterns.md`](docs/patterns.md), the composition cookbook of how anomaly scores plug into the rest of the Prometheus ecosystem (rate-of-change, group rollups, multi-window monitoring, deploy-time silencing, cross-tool joins with promforecast). +[`docs/when-to-use.md`](docs/when-to-use.md) walks through the +"promforecast, promanomaly, or both?" decision tree, and +[`docs/config-schema.md`](docs/config-schema.md) documents the v1 +schema stability commitments. + A self-monitoring example config is bundled at [`examples/configs/self-monitoring.yaml`](examples/configs/self-monitoring.yaml), and a starter `PrometheusRule` at [`examples/alerts/promanomaly-rules.yaml`](examples/alerts/promanomaly-rules.yaml). -Full multi-group production reference configs land in a later release. +The full multi-group production reference deployment — values, NetworkPolicy +egress, ArgoCD `Application`, and Flux `HelmRepository`/`HelmRelease` — +lives at [`examples/production/`](examples/production/). ## Relationship with promforecast @@ -83,9 +90,17 @@ rare events, and per-instance outliers. ## Status -Early development. Output metric names are part of the public API and -follow Prometheus conventions strictly (no colons — those are reserved -for recording rules). +The configuration schema is stable as of the v1.0 release: +`apiVersion: promanomaly.io/v1` and the `values.schema.json` `$id` +both move from alpha to v1. The previous `promanomaly.io/v1alpha1` +alias keeps loading as a byte-identical form with a deprecation +warning. See [`docs/config-schema.md`](docs/config-schema.md) for the +stability commitments, including which changes are breaking and +which are not. + +Output metric names and label sets are part of the public API and +follow Prometheus conventions strictly (no colons — those are +reserved for recording rules). ## Security diff --git a/charts/promanomaly/README.md b/charts/promanomaly/README.md index 231fbee..124046b 100644 --- a/charts/promanomaly/README.md +++ b/charts/promanomaly/README.md @@ -26,8 +26,11 @@ Where ``my-groups.yaml`` carries detector groups under ``groups:``. See ``values.yaml`` and the schema in ``values.schema.json``. The notable bits: -- ``replicaCount`` is pinned to ``1`` — single-replica is the only - supported topology today. HA mode arrives in a later release. +- ``replicaCount`` defaults to ``1``. Single-replica is the simplest + topology; HA mode (``highAvailability.enabled: true``) is supported + and adds Lease-based leader election + a Redis-backed snapshot + cache so multiple replicas serve the same ``/metrics`` snapshot. + See ``examples/configs/ha.yaml`` and ``examples/production/``. - ``existingConfigMap`` lets GitOps tooling (Argo CD, Flux) manage the detector configuration out-of-band; the chart then skips its own ConfigMap rendering. diff --git a/charts/promanomaly/templates/NOTES.txt b/charts/promanomaly/templates/NOTES.txt index 7814709..578b4fa 100644 --- a/charts/promanomaly/templates/NOTES.txt +++ b/charts/promanomaly/templates/NOTES.txt @@ -11,5 +11,19 @@ hot-reloads on change (validation failure rolls back automatically). Datasource: URL: {{ .Values.datasource.url }} +Config schema: this chart renders ``apiVersion: promanomaly.io/v1`` +(stable since the v1.0 release). The legacy ``promanomaly.io/v1alpha1`` +alias still loads as a byte-identical form but emits a deprecation +warning at boot and is scheduled for removal after v2.0. +{{- if .Values.existingConfigMap }} + +NOTE: ``existingConfigMap={{ .Values.existingConfigMap }}`` is set, so the +chart did NOT render its own ConfigMap. If that ConfigMap still declares +``apiVersion: promanomaly.io/v1alpha1`` the pod will boot but will log +``config.api_version_deprecated`` on every restart — bump it to +``promanomaly.io/v1`` (no other changes required). See +docs/config-schema.md for the stability commitment. +{{- end }} + For a quick-start that includes VictoriaMetrics, install ``promanomaly-stack`` instead of this chart. diff --git a/charts/promanomaly/templates/configmap.yaml b/charts/promanomaly/templates/configmap.yaml index 46a4f26..de0239a 100644 --- a/charts/promanomaly/templates/configmap.yaml +++ b/charts/promanomaly/templates/configmap.yaml @@ -13,7 +13,7 @@ metadata: labels: {{- include "promanomaly.labels" . | nindent 4 }} data: config.yaml: | - apiVersion: promanomaly.io/v1alpha1 + apiVersion: promanomaly.io/v1 datasource: url: {{ .Values.datasource.url | quote }} timeout: {{ .Values.datasource.timeout | quote }} diff --git a/charts/promanomaly/values.schema.json b/charts/promanomaly/values.schema.json index 4ad5db1..6b5a4e8 100644 --- a/charts/promanomaly/values.schema.json +++ b/charts/promanomaly/values.schema.json @@ -1,6 +1,8 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://promanomaly.io/schemas/v1/values.schema.json", "title": "promanomaly chart values", + "description": "Stable v1 schema. A future breaking change ships under a fresh $id (e.g. .../v2/values.schema.json) alongside a new apiVersion; this $id remains pinned to the v1 surface for the entire v1.x line.", "type": "object", "additionalProperties": true, "required": ["image", "datasource"], diff --git a/charts/promanomaly/values.yaml b/charts/promanomaly/values.yaml index ab612c4..e79e03a 100644 --- a/charts/promanomaly/values.yaml +++ b/charts/promanomaly/values.yaml @@ -1,6 +1,7 @@ # Default values for the promanomaly detector chart. -# Single-replica only — HA mode lands in a later release and adds -# leader election. See README for the operational story. +# Single-replica by default; HA mode (Lease-based leader election + +# Redis snapshot cache) is opt-in via ``highAvailability.enabled``. +# See README + examples/production/ for the HA topology. image: repository: ghcr.io/esops-dev/promanomaly diff --git a/detector/src/promanomaly/config.py b/detector/src/promanomaly/config.py index 44da33d..399ec00 100644 --- a/detector/src/promanomaly/config.py +++ b/detector/src/promanomaly/config.py @@ -25,8 +25,15 @@ model_validator, ) -CURRENT_API_VERSION = "promanomaly.io/v1alpha1" -SUPPORTED_API_VERSIONS = frozenset({CURRENT_API_VERSION}) +# Graduated from ``v1alpha1`` to ``v1`` in the v1.0 release. The previous +# alpha alias remains valid (byte-identical schema) but loads with a +# ``DeprecationWarning`` so operators get a nudge to bump their configs +# without their pods CrashLoopBackOff'ing on the next image pull. A future +# breaking change ships under a fresh apiVersion (``v2alpha1`` → ``v2``); +# v1 stays the long-term stable surface for the entire v1.x line. +CURRENT_API_VERSION = "promanomaly.io/v1" +DEPRECATED_API_VERSIONS = frozenset({"promanomaly.io/v1alpha1"}) +SUPPORTED_API_VERSIONS = frozenset({CURRENT_API_VERSION}) | DEPRECATED_API_VERSIONS # Prometheus metric/label name. ``id`` and ``group`` are emitted as label # values but must additionally pass the same character rules as metric names @@ -657,6 +664,23 @@ def _validate_api_version(cls, value: str) -> str: if value not in SUPPORTED_API_VERSIONS: supported = sorted(SUPPORTED_API_VERSIONS) raise ValueError(f"unsupported apiVersion {value!r}; expected one of {supported}") + if value in DEPRECATED_API_VERSIONS: + # Byte-identical schema; nudge operators toward ``v1`` so the + # eventual ``v2`` deprecation window doesn't catch them holding + # an alpha alias. ``DeprecationWarning`` here is caught by + # pytest and visible under CLI ``-W default``; the structured + # boot log line is re-emitted from ``serve()`` AFTER + # ``configure_logging`` has wired up the JSON pipeline (see + # ``main.serve``), so operators reading the normal log stream + # see it as a regular JSON warning record. + import warnings + + warnings.warn( + f"apiVersion {value!r} is deprecated; bump configs to " + f"{CURRENT_API_VERSION!r}. The schema is byte-identical.", + DeprecationWarning, + stacklevel=2, + ) return value @field_validator("groups") @@ -730,6 +754,7 @@ def _validate_detector_params(cfg: Config) -> None: __all__ = [ "CURRENT_API_VERSION", + "DEPRECATED_API_VERSIONS", "SUPPORTED_API_VERSIONS", "AlertThresholds", "AuthConfig", diff --git a/detector/src/promanomaly/main.py b/detector/src/promanomaly/main.py index f451ac3..7798418 100644 --- a/detector/src/promanomaly/main.py +++ b/detector/src/promanomaly/main.py @@ -33,7 +33,7 @@ from fastapi import FastAPI from .cache import RedisQueryCache, TTLCache -from .config import Config, load_config +from .config import CURRENT_API_VERSION, DEPRECATED_API_VERSIONS, Config, load_config from .exporter import Exporter, OperationalMetrics from .ha import HAComponents, build_ha_components, run_follower_loop, stop_ha from .leader import resolve_identity @@ -197,6 +197,17 @@ async def reload(self) -> tuple[bool, str]: # changed, mirroring the boot-time first-run pattern. for group in new_config.groups: self._spawn(self._safe_group_run(group.name)) + if new_config.apiVersion in DEPRECATED_API_VERSIONS: + # Mirror the boot-time emission so a hot-reload of a + # legacy-aliased ConfigMap surfaces the same JSON log + # line in the operator's stream — silence after the + # first boot would hide the issue from anyone who + # joined the on-call rotation later. + logger.warning( + "config.api_version_deprecated", + api_version=new_config.apiVersion, + replacement=CURRENT_API_VERSION, + ) logger.info("reload_complete") return True, "ok" @@ -494,6 +505,16 @@ def serve(config_path: str | Path) -> None: groups=[g.name for g in config.groups], refresh_interval_seconds=config.server.refresh_interval_seconds, ) + if config.apiVersion in DEPRECATED_API_VERSIONS: + # Emitted here (not from the pydantic validator) so the warning + # lands in the JSON log stream operators are already tailing. + # The validator additionally fires a ``DeprecationWarning`` for + # CI / pytest visibility. + logger.warning( + "config.api_version_deprecated", + api_version=config.apiVersion, + replacement=CURRENT_API_VERSION, + ) if config.aggressive_refresh: logger.warning( "aggressive_refresh_interval", diff --git a/detector/tests/test_config.py b/detector/tests/test_config.py index abdd541..ae10d55 100644 --- a/detector/tests/test_config.py +++ b/detector/tests/test_config.py @@ -58,6 +58,23 @@ def test_unknown_api_version_rejected() -> None: Config.model_validate(bad) +def test_v1alpha1_api_version_still_loads_with_deprecation_warning() -> None: + # The v1.0 graduation keeps ``v1alpha1`` accepted as a byte-identical + # alias so existing GitOps repos don't CrashLoopBackOff on the image + # bump; the loader nudges operators with a DeprecationWarning. + legacy = {**MIN_CONFIG, "apiVersion": "promanomaly.io/v1alpha1"} + with pytest.warns(DeprecationWarning, match="apiVersion 'promanomaly.io/v1alpha1'"): + cfg = Config.model_validate(legacy) + assert cfg.apiVersion == "promanomaly.io/v1alpha1" + + +def test_current_api_version_is_v1() -> None: + # Pin the graduation: anything other than ``promanomaly.io/v1`` here + # means a release accidentally regressed the schema to alpha or + # leapt to v2 without a deprecation cycle. + assert CURRENT_API_VERSION == "promanomaly.io/v1" + + @pytest.mark.parametrize("bad_id", ["with:colon", "spaces here", "1leading_digit", ""]) def test_invalid_query_id_rejected(bad_id: str) -> None: from pydantic import ValidationError diff --git a/docker/dev-config.yaml b/docker/dev-config.yaml index 8af4489..dde2ed4 100644 --- a/docker/dev-config.yaml +++ b/docker/dev-config.yaml @@ -2,7 +2,7 @@ # scores are produced minutes after the stack starts (instead of after # an hour of warm-up). Not intended for any non-dev use. -apiVersion: promanomaly.io/v1alpha1 +apiVersion: promanomaly.io/v1 datasource: url: http://victoriametrics:8428/ diff --git a/docs/config-schema.md b/docs/config-schema.md new file mode 100644 index 0000000..b3b683e --- /dev/null +++ b/docs/config-schema.md @@ -0,0 +1,58 @@ +# Configuration Schema Stability + +Starting with **v1.0**, promanomaly’s configuration schema is stable. This page explains exactly what that promise means for operators. + +## Versioned Surfaces + +| Surface | Identifier | Current value | +|--------------------------|-------------------------------------|---------------| +| Config API | `apiVersion:` (in YAML) | `promanomaly.io/v1` | +| Helm values schema | `$id` in `values.schema.json` | `https://promanomaly.io/schemas/v1/values.schema.json` | + +A breaking change bumps both versions together. + +## What v1 Guarantees + +**Breaking changes** (require a major version bump + deprecation period): +- Removing or renaming config fields +- Changing default values of fields or detector parameters +- Adding labels to existing output metrics +- Renaming output metrics or labels +- Tightening validation rules + +**Non-breaking changes**: +- Adding new config fields or detector parameters (with sensible defaults) +- Adding new output metrics +- Adding **opt-in** labels (only emitted when the feature is explicitly enabled) +- Adding new detectors +- Loosening validation rules + +**Rule of thumb**: If your existing config still produces the same output metrics with the same labels and the same values after an upgrade, the change is non-breaking. + +## Deprecation Policy + +- Deprecated features keep working for **at least one full minor release**. +- The detector emits clear warnings at startup (structured log + `DeprecationWarning`). +- Helm charts show a NOTES warning during install. +- Every deprecation is listed in `CHANGELOG.md`. + +## Legacy `v1alpha1` Alias + +```yaml +apiVersion: promanomaly.io/v1alpha1 # still works (emits warning) +apiVersion: promanomaly.io/v1 # recommended +``` + +The alias is kept for backward compatibility and will be removed after v2.0. + +## Helm Chart Schema + +The JSON Schema `$id` is stable for the entire v1.x line. Any `# yaml-language-server: $schema=...` pins you have in your `values.yaml` files remain valid across all patch and minor releases. + +## Coordination with promforecast + +promforecast follows the exact same stability rules. Shared concepts (`id`, `group`, and other label contracts) are kept in sync via `LABELS_CONTRACT.md`. + +--- + +This is the complete, operator-friendly reference. Upgrading inside v1.x never requires editing your config files. \ No newline at end of file diff --git a/docs/detectors.md b/docs/detectors.md index f6ef1be..fad5340 100644 --- a/docs/detectors.md +++ b/docs/detectors.md @@ -5,6 +5,23 @@ Every detector is a stateless scorer. It looks at a rolling window of recent sam Sensitivity is controlled globally with `alert_thresholds.score` (default `3.0`). No per-detector tuning knobs. For “which detector should I use?”, see [`decision-tree.md`](decision-tree.md). +For "promforecast vs. promanomaly?", see [`when-to-use.md`](when-to-use.md). + +## How each section is structured + +Every detector page below follows the same five-section template so you +can scan across detectors without re-reading prose for each one: + +1. **What it catches** — the anomaly shapes the detector is designed for. +2. **What it misses** — failure modes the detector is structurally blind + to. Reach for a different detector or compose via PromQL. +3. **How to tune** — the parameters that matter, with sensible default + reasoning. Defaults are part of the public API; changing them is a + breaking change with a deprecation window (see + [`config-schema.md`](config-schema.md)). +4. **When to choose it** — the signal shapes where it's the right pick. +5. **When *not* to choose it** — signal shapes where it's actively the + wrong pick (not just suboptimal). ## Common Output Metrics @@ -22,26 +39,56 @@ detectors: trim_threshold: 3.0 # default ``` -**Best for** -Noisy, heavy-tailed, or non-Gaussian signals: p99 latency, error rates, queue depth, throughput. - -**Strengths** -Extremely robust to outliers by construction — one bad point cannot pull the baseline. - -**Limitations** -- Slowly drifting baselines inside the window → use **Hampel** -- Step changes that have already filled most of the window -- Clustered multi-point anomalies inside the rolling window pollute the median + MAD → enable `recursive_trim` - -**Parameters** -- `recursive_trim` (default `false`) — when true, run the baseline once, drop in-window points above `trim_threshold`, and recompute the median + MAD once more before scoring the latest sample. Bounded to a single trim pass for predictability. Mitigates the clustered-anomaly limitation: a burst of 4–5 anomalous points in the rolling window normally poisons the very median MAD is computed against, and the latest sample looks normal. With `recursive_trim` on, the burst is excluded from the recomputed baseline and the latest sample scores as the operator would expect. -- `trim_threshold` (default `3.0`) — MAD-units cutoff for the trim pass. Matches the conventional 3-MAD anomaly threshold and the default `alert_thresholds.score`. **Deliberately separate from `alert_thresholds.score`** — alerting policy can change without silently re-shaping the baseline. Raise it when the rolling window legitimately contains points beyond 3 MADs that you don't want stripped; lower it for very clean signals where 3 MADs is already noise. +**What it catches.** Single-point and small-burst outliers on noisy, +heavy-tailed, or non-Gaussian signals — p99 latency, error rates, +queue depth, throughput. Robust by construction: one bad point cannot +pull the baseline, so the score on a clean window stays low and the +score on a spike stays high. + +**What it misses.** +- Slowly drifting baselines inside the rolling window — the median + smears the drift and the latest sample stops scoring as anomalous. + Reach for `Hampel`. +- Step changes that have already filled most of the window. By the + time the median has shifted, the change is "the new normal" — reach + for `BOCPD` or `CUSUM`. +- Clustered multi-point anomalies that fill several adjacent samples + poison the median + MAD itself. Enable `recursive_trim` to recover. + +**How to tune.** +- `recursive_trim` (default `false`) — when true, run the baseline + once, drop in-window points above `trim_threshold`, and recompute + the median + MAD once more before scoring the latest sample. + Bounded to a single trim pass for predictability. Mitigates the + clustered-anomaly limitation: a burst of 4–5 anomalous points in + the rolling window normally poisons the very median MAD is + computed against, and the latest sample looks normal. With + `recursive_trim` on, the burst is excluded from the recomputed + baseline and the latest sample scores as the operator would + expect. +- `trim_threshold` (default `3.0`) — MAD-units cutoff for the trim + pass. Matches the conventional 3-MAD anomaly threshold and the + default `alert_thresholds.score`. **Deliberately separate from + `alert_thresholds.score`** — alerting policy can change without + silently re-shaping the baseline. Raise it when the rolling + window legitimately contains points beyond 3 MADs that you don't + want stripped; lower it for very clean signals where 3 MADs is + already noise. Worked example — `clustered_spikes(cluster=6, spike=10.0)`: - Without trim: baseline pulled toward the spike value, score ≈ 0. - With trim: baseline snaps back to the pre-burst mean, score well above 3. - Window size is set globally via `defaults.window`. +Window size is set globally via `defaults.window`. + +**When to choose it.** The default for almost every noisy signal. If +you don't have a strong reason to pick something else, pick this. + +**When *not* to choose it.** Signals where the baseline genuinely +drifts inside the window (use `Hampel`), where the regime steps and +you want fast detection (use `BOCPD` or `CUSUM`), or where the +canonical anomaly is "one member of a fleet looks different" (use +`Cohort`). ## Hampel — Local-window Robust Outlier Filter @@ -54,16 +101,46 @@ detectors: trim_threshold: 3.0 # default ``` -**Best for** -Same signals as MAD, but when the baseline drifts slowly within the rolling window. - -**How it works** -Computes median and MAD over a small local sub-window of `2*t0 + 1` points around the current sample. - -**Parameters** -- `t0` (default `3`) — half-width of the local sub-window. Larger values behave more like classical MAD; smaller values track drift faster. -- `recursive_trim` (default `false`) — when true, recompute the local median + MAD once after excluding sub-window points above `trim_threshold` from the first pass. Same single-iteration bound as MAD. Particularly useful for Hampel because the local sub-window is small: a 4-point cluster right at the tail of the window otherwise fills most of it and the latest sample looks normal. -- `trim_threshold` (default `3.0`) — local-MAD-units cutoff for the trim pass; same default and decoupling rationale as MAD's `trim_threshold`. +**What it catches.** Same signal shapes as MAD — heavy-tailed, +non-Gaussian, noisy — but on signals where the baseline drifts +slowly within the rolling window. Computes median and MAD over a +small local sub-window of `2*t0 + 1` points around the current +sample, so the baseline tracks drift natively instead of being +dragged by the full window. + +**What it misses.** +- Step changes — the local sub-window picks up the new regime + immediately, so the latest sample after a step looks normal. + Reach for `BOCPD` or `CUSUM` for step detection. +- Clustered multi-point anomalies inside the local sub-window + (small by design) poison the local median + MAD. Enable + `recursive_trim` to recover. +- Signals where the baseline is genuinely flat — `MAD` is cheaper + and equivalent. + +**How to tune.** +- `t0` (default `3`) — half-width of the local sub-window. Larger + values behave more like classical MAD; smaller values track drift + faster. +- `recursive_trim` (default `false`) — when true, recompute the + local median + MAD once after excluding sub-window points above + `trim_threshold` from the first pass. Same single-iteration bound + as MAD. Particularly useful for Hampel because the local + sub-window is small: a 4-point cluster right at the tail of the + window otherwise fills most of it and the latest sample looks + normal. +- `trim_threshold` (default `3.0`) — local-MAD-units cutoff for the + trim pass; same default and decoupling rationale as MAD's + `trim_threshold`. + +**When to choose it.** Noisy signals where the baseline is visibly +moving inside your rolling `window` (slow regression, slowly growing +capacity, business-hours ramp) and you want point-level outlier +detection that doesn't fire constantly because the rolling MAD is +chasing the drift. + +**When *not* to choose it.** Stable baselines (use `MAD`), or +step-shifting regimes (use `BOCPD` / `CUSUM`). ### MAD vs Hampel @@ -83,17 +160,36 @@ detectors: alpha: 0.05 # default ``` -**Best for** -Roughly Gaussian signals with a stable but slowly drifting mean: cache hit rate, sustained throughput, slowly growing capacity. - -**How it works** -Maintains an EWMA mean and EWMA variance over the rolling window. Score is the classic z-score: `|y - ewma_mean| / sqrt(ewma_variance)`. The EWMA tracks slow drift natively, so a regression that creeps in over hours does not pollute the baseline the way a fixed-window mean would. - -**Counter-example — do not use on heavy-tailed traffic** -On p99 latency or request rates with fat tails, the variance estimator inflates after the first big spike, the z-score collapses, and you stop catching subsequent anomalies. MAD or Hampel are the right tools for those. - -**Parameters** -- `alpha` (default `0.05`) — EWMA smoothing factor. Smaller values track the mean more slowly; larger values adapt faster but let recent anomalies pollute the baseline more. +**What it catches.** Anomalies on roughly Gaussian signals with a +stable but slowly drifting mean: cache hit rate, sustained +throughput, slowly growing capacity. Maintains an EWMA mean and +EWMA variance over the rolling window; the score is the classic +z-score `|y - ewma_mean| / sqrt(ewma_variance)`. The EWMA tracks +slow drift natively, so a regression that creeps in over hours does +not pollute the baseline the way a fixed-window mean would. + +**What it misses.** +- Heavy-tailed signals. On p99 latency or request rates with fat + tails, the variance estimator inflates after the first big spike, + the z-score collapses, and you stop catching subsequent anomalies. + `MAD` and `Hampel` are the right tools for those. +- Sparse intermittent signals — the variance estimator goes near- + zero between events and the next event scores extreme even when + it's normal for that signal. + +**How to tune.** +- `alpha` (default `0.05`) — EWMA smoothing factor. Smaller values + track the mean more slowly; larger values adapt faster but let + recent anomalies pollute the baseline more. + +**When to choose it.** Smooth, roughly-Gaussian, low-noise signals +where drift is the dominant baseline behaviour — cache hit rate +inching down over the day, capacity slowly growing as you onboard +customers, success rate that's almost flat but not quite. + +**When *not* to choose it.** Anything heavy-tailed (p99 latency, +error rates), bursty traffic (request rates with diurnal pattern), +or sparse counters (rare events). Reach for `MAD` or `IQR`. ## IQR — Tukey Interquartile Range @@ -104,14 +200,30 @@ detectors: k: 1.5 # default ``` -**Best for** -Non-parametric detection with a sensitivity profile that differs from MAD — lighter on the tails, more weight on the middle 50% of the window. Pairs well with MAD inside an ensemble because the two disagree in interesting ways on heavy-tailed signals. +**What it catches.** Non-parametric outliers with a sensitivity +profile that differs from MAD — lighter on the tails, more weight +on the middle 50% of the window. Computes `Q1` and `Q3` and flags +samples outside `[Q1 - k*IQR, Q3 + k*IQR]`; the score is the +normalised distance outside the band, in IQR units. + +**What it misses.** +- Slow baseline drift (use `Hampel` or `ZScoreEWMA`). +- Clustered multi-point anomalies that shift `Q1` or `Q3` — IQR has + no `recursive_trim` equivalent, so a sustained burst can pull the + quartiles enough to silence the latest sample. -**How it works** -Computes the first and third quartiles (`Q1`, `Q3`) and flags samples outside `[Q1 - k*IQR, Q3 + k*IQR]`. Score is the normalised distance outside the band, in IQR units. +**How to tune.** +- `k` (default `1.5`) — multiplier for the IQR band. `1.5` is + Tukey's classical "outlier" definition; `3.0` is "far outlier". -**Parameters** -- `k` (default `1.5`) — multiplier for the IQR band. `1.5` is Tukey's classical "outlier" definition; `3.0` is "far outlier". +**When to choose it.** As the second detector in a `voting` ensemble +with `MAD`. The two disagree in interesting ways on heavy-tailed +signals, and `voting: min_detectors=2` cuts the false-positive rate +of either detector alone without losing real anomalies. + +**When *not* to choose it.** As the only detector on a heavy-tailed +signal — `MAD` is the better default. As a primary detector for +drift (use `Hampel` / `ZScoreEWMA`). ### MAD vs IQR sensitivity @@ -132,24 +244,62 @@ detectors: bucket_width: 1h # default ``` -**Best for** -Signals with strong diurnal patterns where a fixed-window MAD smears the daily shape and either false-positives every off-peak hour or false-negatives genuine anomalies. The canonical case: "Tuesday 3am looks unlike prior 3am samples." +**What it catches.** Anomalies on signals with strong diurnal +patterns where a fixed-window MAD smears the daily shape and either +false-positives every off-peak hour or false-negatives genuine +anomalies. The canonical case: "Tuesday 3am looks unlike prior 3am +samples." Buckets every sample in the lookback window by its UTC +hour-of-day (24 buckets by default), then scores the latest sample +against the same-bucket median and MAD. The runner automatically +expands the per-query fetch window, so it composes with short-window +detectors (`MAD` / `Hampel`) in the same query without forcing the +group's `defaults.window` to grow. **This is still baseline detection, not forecasting.** -The detector scores the present against historical observations from the same hour-of-day bucket. It never predicts future values — that's promforecast's job. - -**How it works** -Bucket every sample in the lookback window by its UTC hour-of-day (24 buckets by default). For the latest sample, compute the median and MAD across the same-bucket history and score it the same way the standard MAD detector does. The runner automatically expands the per-query fetch window when HourOfDayMAD is configured, so it composes with short-window detectors (MAD / Hampel) in the same query without forcing the group's `defaults.window` to grow. - -**Parameters** -- `lookback` (default `7d`) — how far back to fetch baseline samples. -- `bucket_width` (default `1h`) — must divide 24h evenly (e.g. `30m`, `1h`, `2h`, `4h`). - -**Time zone caveat** -Buckets are computed against **UTC** wall-clock time because Prometheus stores UTC. If your operations team thinks in a non-UTC timezone (e.g. JST = UTC+9), "9 AM local" buckets line up with UTC midnight bucket; bucket boundaries don't align with local "hour of day". This is fine for periodic anomaly detection but can confuse on-call ("the alert says hour 0 was anomalous; it's 9 AM here"). A local-timezone option is deferred until enough operators ask for it — for now, document the offset alongside your runbook or use a `bucket_width` that's a divisor of your timezone offset to keep buckets aligned. - -**Resource note** -A long lookback against a large query can pull a lot of data; respect `safety.max_series_per_query` and `safety.max_total_series`. The full day-of-week / multi-week version is `DayOfWeekMAD` (below) and uses the runner's sliding-window fetch strategy to keep TSDB load bounded. +The detector scores the present against historical observations +from the same hour-of-day bucket. It never predicts future values — +that's promforecast's job. + +**What it misses.** +- Weekly periodicity. If weekday vs. weekend behaviour differs + materially, the per-hour median collapses across both and you'll + either over-fire on weekends or miss weekday anomalies. Use + `DayOfWeekMAD` instead. +- Sub-hour fluctuations inside a bucket — by design, the bucket + width is the granularity. + +**How to tune.** +- `lookback` (default `7d`) — how far back to fetch baseline + samples. Sized to give every bucket ≥ 7 observations to median + over. +- `bucket_width` (default `1h`) — must divide 24h evenly (e.g. + `30m`, `1h`, `2h`, `4h`). Smaller buckets sharpen the diurnal + shape but each bucket has fewer observations. + +**Time zone caveat.** Buckets are computed against **UTC** wall-clock +time because Prometheus stores UTC. If your operations team thinks +in a non-UTC timezone (e.g. JST = UTC+9), "9 AM local" buckets line +up with UTC midnight bucket; bucket boundaries don't align with +local "hour of day". This is fine for periodic anomaly detection but +can confuse on-call ("the alert says hour 0 was anomalous; it's 9 AM +here"). A local-timezone option is deferred until enough operators +ask for it — for now, document the offset alongside your runbook or +use a `bucket_width` that's a divisor of your timezone offset to +keep buckets aligned. + +**Resource note.** A long lookback against a large query can pull a +lot of data; respect `safety.max_series_per_query` and +`safety.max_total_series`. The full day-of-week / multi-week version +is `DayOfWeekMAD` (below) and uses the runner's sliding-window fetch +strategy to keep TSDB load bounded. + +**When to choose it.** Diurnal signals whose daily shape is +consistent across the week — request rates, batch counters, business +KPIs where Monday's 3am ≈ Tuesday's 3am ≈ Sunday's 3am. + +**When *not* to choose it.** Weekly-shaped signals (use +`DayOfWeekMAD`), signals without strong periodicity (use plain +`MAD`), or signals whose daily shape itself changes (use `BOCPD`). ## DayOfWeekMAD — Stratified by (Day-of-Week, Hour-of-Day) @@ -162,36 +312,87 @@ detectors: baseline_refresh_interval: 1h # default ``` -**Best for** -Weekly-periodic signals where weekend traffic behaves nothing like weekday traffic — B2B request rates that go quiet on Saturdays, batch jobs that only run on Mondays, business-hours error rates that differ from overnight. A single `HourOfDayMAD` median-collapses weekday + weekend samples and either smears the baseline or false-positives every weekend afternoon; `DayOfWeekMAD` keeps them separate via a 7x24 grid. +**What it catches.** Anomalies on weekly-periodic signals where +weekend traffic behaves nothing like weekday traffic — B2B request +rates that go quiet on Saturdays, batch jobs that only run on +Mondays, business-hours error rates that differ from overnight. +Buckets every sample in the lookback window by +`(weekday(), hour-of-day)` (default 7x24 grid, all in UTC) and +scores the latest sample against the same-bucket median + MAD. The +`weekday()` value follows Python's `datetime.weekday()` convention +(Monday=0). **This is still baseline detection, not forecasting.** -Same carve-out as `HourOfDayMAD`. The detector scores the present against historical observations from the same `(day-of-week, hour-of-day)` bucket — it never predicts future values. - -**How it works** -Bucket every sample in the lookback window by `(weekday(), hour-of-day)` (default 7x24 grid, all in UTC). For the latest sample, find its bucket, compute median + MAD over the same-bucket history, and score `|y - bucket_median| / bucket_MAD`. The `weekday()` value follows Python's `datetime.weekday()` convention (Monday=0). - -**Parameters** -- `lookback` (default `4w`) — multi-week window. Sized in weeks so every bucket has several observations to median over. Accepts `w` (weeks), `d` (days), `h` (hours), `m` (minutes). -- `bucket_width` (default `1h`) — within-day bucket width; must divide 24h evenly. -- `baseline_refresh_interval` (default `1h`) — opts the detector into the runner's sliding-window fetch strategy: the heavy multi-week query runs at this cadence rather than every `server.refresh_interval`. See [`scaling-stratified.md`](scaling-stratified.md) for the operational rationale. - -**Time zone caveat** -Both `weekday()` and the within-day bucket are computed in **UTC**. For operators in non-UTC timezones the weekday boundary lands at local times other than midnight: in JST (UTC+9), the UTC "Sunday" bucket covers Sunday 09:00 → Monday 09:00 local. That's fine for weekly periodicity detection but operators reading dashboards should remember that "weekday 5 (Saturday)" in a metric label refers to UTC Saturday, not local Saturday. Same deferred-timezone-option rationale as `HourOfDayMAD`. - -**When to choose** -- Use `DayOfWeekMAD` when weekday vs. weekend (or Monday-vs-rest) behaviour materially differs. -- Use `HourOfDayMAD` when the daily shape is consistent across the week and a one-week lookback is enough. -- Use plain `MAD` when neither dimension is informative — the rolling window already captures the relevant context. - -**Example** -[`examples/configs/weekly-baseline.yaml`](../examples/configs/weekly-baseline.yaml) ships a production-shaped reference combining `HourOfDayMAD` + `DayOfWeekMAD` + `MAD` with a voting ensemble — a useful starting point if your signal has both daily and weekly structure. - -**Resource note** -The sliding-window strategy keeps TSDB load bounded even at a 4w lookback: the long query fires once per `baseline_refresh_interval` (default `1h`) while the latest sample is still fetched every refresh and stitched onto the cached baseline before scoring. `validate --probe` reports the projected per-day query count for each stratified detector — see [`scaling-stratified.md`](scaling-stratified.md). - -**Calibration caveat** -The `auto_select` cycle and the `anomaly_confidence_score` / `anomaly_baseline_stability` gauges were designed for rolling-window detectors and are imperfect for stratified ones — the synthetic-injection patterns partially contaminate same-bucket baselines, and the stability proxy compares bucket medians that legitimately differ on a clean signal. For stratified-only configs, prefer explicit detector selection (`auto_select: false`) and consider `defaults.emit_baseline_stability: false`. Full rationale + a future-fix note in [`scaling-stratified.md`](scaling-stratified.md#calibration-confidence-and-auto-select-caveats). +Same carve-out as `HourOfDayMAD`. The detector scores the present +against historical observations from the same `(day-of-week, +hour-of-day)` bucket — it never predicts future values. + +**What it misses.** +- Anomalies that span a full bucket — if Sunday afternoons are + always slightly off, the median already encodes "slightly off" + and a slightly-more-off Sunday won't score. +- Recent regime shifts (last week was a new product launch and + this week's baseline is the new normal) — the multi-week median + is dragging old behaviour. Pair with `BOCPD` for that case. +- Sub-bucket fluctuations — by design. + +**How to tune.** +- `lookback` (default `4w`) — multi-week window. Sized in weeks so + every bucket has several observations to median over. Accepts `w` + (weeks), `d` (days), `h` (hours), `m` (minutes). +- `bucket_width` (default `1h`) — within-day bucket width; must + divide 24h evenly. +- `baseline_refresh_interval` (default `1h`) — opts the detector + into the runner's sliding-window fetch strategy: the heavy + multi-week query runs at this cadence rather than every + `server.refresh_interval`. See + [`scaling-stratified.md`](scaling-stratified.md) for the operational + rationale. + +**Time zone caveat.** Both `weekday()` and the within-day bucket are +computed in **UTC**. For operators in non-UTC timezones the weekday +boundary lands at local times other than midnight: in JST (UTC+9), +the UTC "Sunday" bucket covers Sunday 09:00 → Monday 09:00 local. +That's fine for weekly periodicity detection but operators reading +dashboards should remember that "weekday 5 (Saturday)" in a metric +label refers to UTC Saturday, not local Saturday. Same +deferred-timezone-option rationale as `HourOfDayMAD`. + +**Resource note.** The sliding-window strategy keeps TSDB load +bounded even at a 4w lookback: the long query fires once per +`baseline_refresh_interval` (default `1h`) while the latest sample +is still fetched every refresh and stitched onto the cached baseline +before scoring. `validate --probe` reports the projected per-day +query count for each stratified detector — see +[`scaling-stratified.md`](scaling-stratified.md). + +**Calibration caveat.** The `auto_select` cycle and the +`anomaly_confidence_score` / `anomaly_baseline_stability` gauges +were designed for rolling-window detectors and are imperfect for +stratified ones — the synthetic-injection patterns partially +contaminate same-bucket baselines, and the stability proxy compares +bucket medians that legitimately differ on a clean signal. For +stratified-only configs, prefer explicit detector selection +(`auto_select: false`) and consider +`defaults.emit_baseline_stability: false`. Full rationale + a +future-fix note in +[`scaling-stratified.md`](scaling-stratified.md#calibration-confidence-and-auto-select-caveats). + +**When to choose it.** Weekly-periodic signals where weekday and +weekend look genuinely different — B2B traffic, business KPIs, +batch-style workloads. The combined daily + weekly stratification +keeps the baseline tight where `HourOfDayMAD` would smear. + +**When *not* to choose it.** Signals whose daily shape is the same +across the week (use `HourOfDayMAD` — cheaper and a 1w lookback is +enough), signals without seasonality (use plain `MAD`), or signals +whose seasonal shape itself shifts frequently (use `BOCPD`). + +**Example.** +[`examples/configs/weekly-baseline.yaml`](../examples/configs/weekly-baseline.yaml) +ships a production-shaped reference combining `HourOfDayMAD` + +`DayOfWeekMAD` + `MAD` with a voting ensemble — a useful starting +point if your signal has both daily and weekly structure. ## BOCPD — Bayesian Online Change-Point Detection @@ -204,19 +405,52 @@ detectors: lag: 15 # default ``` -**Best for** -Detecting regime shifts on signals where you want a probabilistic change-point score — deploys, rolling restarts, autoscaler steps, anything that breaks the baseline rather than spiking through it. - -**How it works** -Models the observation stream with a Normal-Gamma conjugate prior and maintains a posterior distribution over the *run length* (samples since the last change-point). The score is the posterior mass at recent run lengths (`r_t <= lag`) — high when a change-point happened in the last `lag` samples. Fires when score ≥ `threshold`; `threshold` is a per-detector override of `alert_thresholds.score` because the score is a probability, not a sigma multiplier. - -**Parameters** -- `hazard` (default `250`) — expected interval between change-points in samples. -- `threshold` (default `0.5`) — posterior probability above which the detector fires. -- `lag` (default `15`) — inclusive run-length window counted as "recent". - -**Trade-offs** -Probabilistic and confidence-gateable; more expensive than CUSUM (`O(window²)` per fit, still cheap in absolute terms); assumes Gaussian-ish observations. +**What it catches.** Regime shifts on signals where you want a +probabilistic change-point score — deploys, rolling restarts, +autoscaler steps, anything that breaks the baseline rather than +spiking through it. Models the observation stream with a +Normal-Gamma conjugate prior and maintains a posterior distribution +over the *run length* (samples since the last change-point). The +score is the posterior mass at recent run lengths +(`r_t <= lag`) — high when a change-point happened in the last +`lag` samples. Fires when score ≥ `threshold`; `threshold` is a +per-detector override of `alert_thresholds.score` because the score +is a probability, not a sigma multiplier. + +**What it misses.** +- Point spikes — BOCPD scores low on isolated outliers that don't + shift the regime. Pair with `MAD` for that. +- Strongly non-Gaussian distributions. The Normal-Gamma prior + assumes Gaussian-ish observations; heavy-tailed signals produce + noisy change-point probabilities. +- Periodic anomalies that look like regime shifts to BOCPD — the + posterior keeps re-firing every period. + +**How to tune.** +- `hazard` (default `250`) — expected interval between change-points + in samples. Raise on signals where breaks are rare (lower + false-positive rate but slower detection); lower for chatty + environments. +- `threshold` (default `0.5`) — posterior probability above which + the detector fires. The probability scale means thresholds above + `0.7` are very confident; below `0.3` is essentially random. +- `lag` (default `15`) — inclusive run-length window counted as + "recent". Larger values catch slower transitions; smaller values + fire only on abrupt shifts. + +**Trade-offs.** Probabilistic and confidence-gateable; more +expensive than CUSUM (`O(window²)` per fit, still cheap in absolute +terms); assumes Gaussian-ish observations. + +**When to choose it.** Step-change detection where you want a +confidence-gated alert: "fire when the posterior probability that +the regime just shifted is ≥ 0.7". Useful when you want to +distinguish "still anomalous" from "regime just shifted" via the +composition recipe in [`patterns.md`](patterns.md#change-points-composed-with-outlier-alerts). + +**When *not* to choose it.** Point-spike detection (use `MAD` / +`Hampel`), or heavy-tailed signals where the Normal-Gamma prior +mis-fits. For simpler step detection, `CUSUM` is faster. ## CUSUM — Cumulative Sum @@ -228,18 +462,41 @@ detectors: h: 5.0 # default ``` -**Best for** -A fast, interpretable change-point detector. Workhorse of industrial process control; pairs well with BOCPD when you want both probabilistic and classical signals on the same metric. - -**How it works** -Tracks running upper/lower cumulative sums of `(x - median - k*sigma)` against a MAD-derived sigma estimate. A change-point fires on a *fresh* crossing of `h*sigma` at the latest sample (no double-counting on overlapping rolling-window calls). The score is `max(|S+|, |S-|) / sigma`, in sigma units — same scale as MAD/Hampel. - -**Parameters** -- `k` (default `0.5`) — reference value (in sigma units); slack subtracted before accumulation. -- `h` (default `5.0`) — decision threshold (in sigma units). - -**Trade-offs** -Single-pass, `O(window)`; intuitive sigma-multiplier units; less natural for non-Gaussian likelihoods but the MAD-derived sigma helps on heavy-tailed signals. +**What it catches.** Fast, classical step-change detection. Tracks +running upper/lower cumulative sums of `(x - median - k*sigma)` +against a MAD-derived sigma estimate. A change-point fires on a +*fresh* crossing of `h*sigma` at the latest sample (no +double-counting on overlapping rolling-window calls). The score is +`max(|S+|, |S-|) / sigma`, in sigma units — same scale as MAD/Hampel. + +**What it misses.** +- Point spikes — same blind spot as BOCPD; pair with `MAD`. +- Gradual drift that never accumulates enough sigma to cross `h` — + reach for `Hampel` or `ZScoreEWMA`. +- Probabilistic interpretation — `CUSUM` returns a sigma multiplier, + not a probability. Use `BOCPD` when you want + `anomaly_confidence_score`-style gating. + +**How to tune.** +- `k` (default `0.5`) — reference value (in sigma units); slack + subtracted before accumulation. Higher `k` ignores smaller drifts; + lower `k` fires on subtler shifts. +- `h` (default `5.0`) — decision threshold (in sigma units). The + conservative default keeps the false-positive rate low; halve it + on signals where you want to catch smaller-magnitude shifts. + +**Trade-offs.** Single-pass, `O(window)`; intuitive sigma-multiplier +units; less natural for non-Gaussian likelihoods but the +MAD-derived sigma helps on heavy-tailed signals. + +**When to choose it.** Fast, time-tested step-change detection for +operators who don't need a probabilistic score. The workhorse of +industrial process control; pairs well with `BOCPD` when you want +both probabilistic and classical signals on the same metric. + +**When *not* to choose it.** When you want a probability-scale +output (use `BOCPD`), gradual-drift detection (use `Hampel` / +`ZScoreEWMA`), or point-spike outlier detection (use `MAD`). ### When to choose BOCPD vs CUSUM @@ -263,42 +520,108 @@ detectors: min_cohort_size: 5 # default; cohorts smaller than this are skipped ``` -**Best for** -Fleets of nominally identical workloads — nodes, pods, replicas of a service — where the anomaly only appears when a member is compared against its peers. Each individual series can look perfectly normal in its own rolling window; the divergence is the signal. - -**How it works** -The runner groups every series returned by the query into **cohorts** — series sharing every label except `cohort_label`. For each cohort with at least `min_cohort_size` members: - -1. At every timestamp `t` in the rolling window, compute the cross-member median and MAD of the values. -2. Summarise to a single `(cohort_median, cohort_mad)` by taking the median over time of each per-step statistic (which keeps the baseline insensitive to short bursts in one member). -3. For each member, score `|rolling_mean(member) - cohort_median| / cohort_mad` — the member's typical level expressed in cohort-MAD units. - -Cohorts below `min_cohort_size` are skipped silently: members receive a zero-score row but no firing. The detector emits an opt-in `cohort_label=""` output label so dashboards can pivot on the chosen grouping. - -**Parameters** -- `cohort_label` (default `"instance"`) — the label that varies between cohort members. Common choices: `instance`, `pod`, `replica`. Every *other* label on the series identifies the cohort itself. -- `min_cohort_size` (default `5`) — minimum members required for scoring. Guards against the degenerate two-member case where one member is the cohort's own median by definition. - -**Trade-offs** -- Catches the "one bad node in the fleet" case that no single-series detector can. -- **Not** multivariate / covariance-shift analysis — that lives in the roadmap's future-considerations list, not in Cohort itself. Cohort comparison is the restricted, well-understood version of the problem. -- Calibration / `auto_select` are designed around single-series detectors. The runner **skips the calibration cycle entirely for cohort-aware detectors**, so `anomaly_confidence_score` and `anomaly_baseline_stability` are simply not emitted for Cohort. Pair Cohort with explicit detector selection rather than `auto_select`, and prefer a dedicated group when mixing it with single-series detectors so the calibration gauges for the latter aren't blocked by Cohort's presence. The runner emits a one-time boot warning (`cohort_with_auto_select_true`) when it detects a Cohort plan in a group with `auto_select: true`. -- `/debug/inspect` and the `inspect` CLI build the same per-cohort baseline the runner uses, so inspecting one member shows the real cross-member score — not the zero a naive single-series inspection would return. - -**Edge cases worth knowing** - -- **Constant fleet (cohort MAD = 0).** When most cohort members sit on the same exact value, the median absolute deviation collapses to zero and the score collapses to zero too. This matches the MAD detector's behaviour on a flatline window and biases toward false negatives, not false positives. Cohort is built for **noisy continuous signals** (CPU, latency, throughput). For discrete or quantised signals (`up{}`, integer counters with little variation), pair Cohort with PromQL preprocessing (e.g. `rate()`, `irate()`) that introduces meaningful variation, or use a different detector on a derived signal. -- **Self-inclusion in the baseline.** The cross-cohort median and MAD are computed over **every** member, including the one being scored. For typical fleet sizes (≥ 10) this bias is negligible; for the minimum-size case (`min_cohort_size: 5`) a single outlier nudges the baseline toward itself and slightly shrinks its own score. The current implementation accepts this for simplicity — leave-one-out would require recomputing N baselines per query and the operational gain is small at the fleet sizes Cohort targets. -- **Cohort + `discover:`.** When `discover.variable` and `cohort_label` reference the **same** label, each discover-rendered query receives only one member per cohort — every cohort falls below `min_cohort_size` and Cohort emits zero-score rows for all members. Either pick a `cohort_label` that's *not* one of the discovered variables, or drop the `discover:` block and let the cohort form from a single PromQL fetch. - -**Worked example — `cohort_label: instance` across six nodes** - -Given six series sharing `job=node` and differing only on `instance`, where five hover at ~10 and one drifts to ~30: +**What it catches.** Anomalies in fleets of nominally identical +workloads — nodes, pods, replicas of a service — where the anomaly +only appears when a member is compared against its peers. Each +individual series can look perfectly normal in its own rolling +window; the divergence is the signal. The runner groups every series +returned by the query into **cohorts** — series sharing every label +except `cohort_label`. For each cohort with at least +`min_cohort_size` members: + +1. At every timestamp `t` in the rolling window, compute the + cross-member median and MAD of the values. +2. Summarise to a single `(cohort_median, cohort_mad)` by taking the + median over time of each per-step statistic (which keeps the + baseline insensitive to short bursts in one member). +3. For each member, score + `|rolling_mean(member) - cohort_median| / cohort_mad` — the + member's typical level expressed in cohort-MAD units. + +Cohorts below `min_cohort_size` are skipped silently: members +receive a zero-score row but no firing. The detector emits an opt-in +`cohort_label=""` output label so dashboards can pivot on the +chosen grouping. + +**What it misses.** +- True multivariate / covariance-shift effects across multiple + signals — that's a different (and harder) problem; deferred to v2. +- Anomalies where the whole fleet shifts together. By construction, + Cohort is blind to a regression that affects every member equally. + Pair Cohort with a single-series detector in the same query to + cover both shapes. +- Constant fleets (cohort MAD = 0). When most cohort members sit on + the same exact value, the median absolute deviation collapses to + zero and the score collapses to zero too. Cohort is built for + **noisy continuous signals** (CPU, latency, throughput); use + PromQL preprocessing (`rate()`, `irate()`) on discrete signals. +- The "one is the cohort median by definition" case at very small + cohort sizes — guard with `min_cohort_size: 5` or higher. + +**How to tune.** +- `cohort_label` (default `"instance"`) — the label that varies + between cohort members. Common choices: `instance`, `pod`, + `replica`. Every *other* label on the series identifies the cohort + itself. +- `min_cohort_size` (default `5`) — minimum members required for + scoring. Guards against the degenerate two-member case where one + member is the cohort's own median by definition. Raise to 10+ on + large fleets where self-inclusion bias is a concern. + +**Trade-offs.** +- Calibration / `auto_select` are designed around single-series + detectors. The runner **skips the calibration cycle entirely for + cohort-aware detectors**, so `anomaly_confidence_score` and + `anomaly_baseline_stability` are simply not emitted for Cohort. + Pair Cohort with explicit detector selection rather than + `auto_select`, and prefer a dedicated group when mixing it with + single-series detectors so the calibration gauges for the latter + aren't blocked by Cohort's presence. The runner emits a one-time + boot warning (`cohort_with_auto_select_true`) when it detects a + Cohort plan in a group with `auto_select: true`. +- `/debug/inspect` and the `inspect` CLI build the same per-cohort + baseline the runner uses, so inspecting one member shows the real + cross-member score — not the zero a naive single-series inspection + would return. + +**Edge cases worth knowing.** +- **Self-inclusion in the baseline.** The cross-cohort median and + MAD are computed over **every** member, including the one being + scored. For typical fleet sizes (≥ 10) this bias is negligible; + for the minimum-size case (`min_cohort_size: 5`) a single outlier + nudges the baseline toward itself and slightly shrinks its own + score. The current implementation accepts this for simplicity — + leave-one-out would require recomputing N baselines per query and + the operational gain is small at the fleet sizes Cohort targets. +- **Cohort + `discover:`.** When `discover.variable` and + `cohort_label` reference the **same** label, each discover-rendered + query receives only one member per cohort — every cohort falls + below `min_cohort_size` and Cohort emits zero-score rows for all + members. Either pick a `cohort_label` that's *not* one of the + discovered variables, or drop the `discover:` block and let the + cohort form from a single PromQL fetch. + +**Worked example — `cohort_label: instance` across six nodes.** +Given six series sharing `job=node` and differing only on `instance`, +where five hover at ~10 and one drifts to ~30: - Cohort median converges to ~10, cohort MAD to ~0.1. -- Score for the divergent member is `|30 - 10| / 0.1 = 200` — well above the default alert threshold of 3. +- Score for the divergent member is `|30 - 10| / 0.1 = 200` — well + above the default alert threshold of 3. - Scores for the five clean members are near zero. -See [`patterns.md`](patterns.md) for composing Cohort output with other detectors via PromQL joins on `(id, group, instance)`. +**When to choose it.** Identical-workload fleets — node-exporter +targets, kube-state-metrics per-pod data, per-replica request +latency — where "one is misbehaving compared to its peers" is the +canonical anomaly shape. The case no single-series detector can see. + +**When *not* to choose it.** Single-series signals (use any of the +single-series detectors), discrete / quantised signals without +PromQL preprocessing, or fleets where the canonical anomaly is the +whole fleet shifting together (pair with a single-series detector +on a `sum`/`avg`-aggregated query, or skip Cohort entirely). + +See [`patterns.md`](patterns.md) for composing Cohort output with +other detectors via PromQL joins on `(id, group, instance)`. ## Auto-select Best Detector Per Series diff --git a/docs/patterns.md b/docs/patterns.md index b21f471..6036e1c 100644 --- a/docs/patterns.md +++ b/docs/patterns.md @@ -2,6 +2,15 @@ Most advanced use cases are solved by combining promanomaly’s plain Prometheus metrics (`anomaly_score`, `anomaly_outside_threshold`, etc.) with standard PromQL, recording rules, or Alertmanager. No new detector features required. +This page is the **composition cookbook** for everything that looks +like a missing feature but is actually a recurring pattern. If you +catch yourself wishing for a new config knob or a new detector type, +check here first — there's a good chance the recipe already exists. + +The features deliberately kept out of promanomaly core (and routed to +composition instead) are documented at the end under +[**Why each of these lives here, not in the detector**](#why-each-of-these-lives-here-not-in-the-detector). + ## Rate-of-Change & Acceleration Detect anomalies in *how fast* a metric changes, not its absolute value. @@ -178,4 +187,129 @@ Do NOT silence change-point alerts during deploys inside the detector — fire o (anomaly_outside_threshold == 1) ``` -Both tools follow the same `id` + `group` label contract. +Both tools follow the same `id` + `group` label contract — see +[`when-to-use.md`](when-to-use.md) for the decision tree and +[`LABELS_CONTRACT.md`](../LABELS_CONTRACT.md) for the exact label +guarantee. Either direction of the join is meaningful: requiring +both means "slow capacity drift AND a fast-moving anomaly on the same +signal right now"; an `or` rollup means "anything wrong with this +signal". + +A useful production pattern is one recording rule for each: + +```yaml +- record: signal_in_trouble:any + expr: | + forecast_deviation_outside_band == 1 + or on (id, group) anomaly_outside_threshold == 1 + +- record: signal_in_trouble:both + expr: | + forecast_deviation_outside_band == 1 + and on (id, group) anomaly_outside_threshold == 1 +``` + +…and an Alertmanager `inhibition_rule` so an `:both` firing +suppresses the noisier `:any`. + +## Deploy-Time Silencing + +Anomaly scores should never be damped, masked, or paused during +deploys, rolling restarts, or maintenance windows — real outages +happen during those windows precisely because deploys cause them. +Silencing belongs in Alertmanager, fired by a webhook from your +deploy pipeline: + +```bash +# In your CD pipeline, before the rollout: +curl -X POST "$AM/api/v2/silences" -d '{ + "matchers": [ + {"name": "id", "value": "checkout_p95", "isRegex": false} + ], + "startsAt": "'"$(date -u +%FT%TZ)"'", + "endsAt": "'"$(date -u -d '+30 min' +%FT%TZ)"'", + "createdBy": "ci-bot", + "comment": "checkout deploy $CI_PIPELINE_URL" +}' +``` + +The detector keeps emitting `anomaly_score` and +`anomaly_outside_threshold` exactly as before; Alertmanager just +silently drops the resulting alert for the silence's lifetime. The +signal stays visible on Grafana for post-incident analysis without +firing a page. **This is the correct way to do it** — see +[**Why each of these lives here, not in the detector**](#why-each-of-these-lives-here-not-in-the-detector) +for the rationale. + +## Confidence-Gated Alerts + +When auto-select or stratified detectors are emitting +`anomaly_confidence_score` and `anomaly_baseline_stability`, gate +alerts on both signals so warm-up windows and calibration noise +don't page: + +```promql +anomaly_outside_threshold == 1 + and on (id, group, detector) anomaly_confidence_score > 0.7 + and on (id, group, detector) anomaly_baseline_stability > 0.6 +``` + +The bundled `AnomalyConfidenceLow` PrometheusRule does the inverse +side of the same gate (alert on low confidence on a firing detector +so you find out the calibration is degrading before you stop +trusting the score). See [`confidence.md`](confidence.md) for the +underlying formulas. + +## Practical-Significance Floors + +Statistical significance ≠ operational significance. A 3-MAD move on +a counter that went from `0.01` errors/s to `0.012` errors/s is +real, but probably not page-worthy. Set a floor: + +```yaml +defaults: + min_abs_delta: 0.05 # 0.05 errors/s minimum before + # anomaly_outside_threshold can flip + min_relative_delta: 0.05 # OR 5% of baseline +``` + +`anomaly_score` is unaffected — dashboards still see the raw signal. +Only the `anomaly_outside_threshold` bit is gated. Tune per-query +where the group-level floor is wrong: + +```yaml +queries: + - id: chatty_signal + promql: ... + min_abs_delta: 0.5 # noisier signal needs a higher floor + detectors: + - name: MAD +``` + +## Why each of these lives here, not in the detector + +A recurring pattern in design review is "this should be a feature +flag in the detector". Most of the time, the answer is that PromQL, +recording rules, and Alertmanager already cover it — and growing the +detector to cover it duplicates ecosystem tooling poorly. The +features kept out of core, with the composition recipe that replaces +them: + +| Asked for | Lives instead in | Recipe | +|---|---|---| +| Rate-of-change / acceleration detector | PromQL `rate()` / `deriv()` wrapper on the source query | [Rate-of-Change & Acceleration](#rate-of-change--acceleration) | +| Group / fleet rollup metrics | Recording rules over `anomaly_*` | [Group-Level Rollups](#group-level-rollups) | +| `for:`-style hysteresis inside the detector | Alertmanager `for:` on the firing rule | [Flap Suppression](#flap-suppression) | +| Multi-window detection as a config knob | Repeat the detector with `instance:` | [Multi-Window Detection](#multi-window-detection) | +| Per-detector silencing windows / deploy-aware damping | Alertmanager silences, fired by webhook | [Deploy-Time Silencing](#deploy-time-silencing) | +| Weighted / Bayesian / ML ensemble fusion | Compose `anomaly_outside_threshold` via recording rules | [Group-Level Rollups](#group-level-rollups) | +| "Known-noisy series" mute lists | Raise the threshold per query, swap detector, or drop the series from config | (no recipe — these are tuning, not silencing) | +| Built-in alerting policy (severity escalations, "page only on second occurrence") | Alertmanager routes + grouping | (Alertmanager docs) | +| Per-query post-processing PromQL | Recording rules on the emitted anomaly metrics | (Prometheus recording-rule docs) | +| Built-in UI | Grafana with the bundled dashboards | [`dashboards/grafana/`](../dashboards/grafana/) | + +The detector's job is **emitting the signal**. Anything that looks +like alerting policy, silencing, severity, or rollup belongs in the +surrounding ecosystem. Pushing those features back into composition +recipes is how promanomaly stays small, stateless, and +ecosystem-friendly. diff --git a/docs/when-to-use.md b/docs/when-to-use.md new file mode 100644 index 0000000..5238e61 --- /dev/null +++ b/docs/when-to-use.md @@ -0,0 +1,96 @@ +# When to Use promforecast, promanomaly, or Both + +The two tools are **complementary**, not overlapping. This guide helps you pick the right one (or both) for any signal before you start tuning. + +For “which detector inside promanomaly?” see [`decision-tree.md`](decision-tree.md). +For composition recipes (rate-of-change, multi-window, cross-tool joins) see [`patterns.md`](patterns.md). + +## One-Line Summary + +- **promforecast** answers: *“What will this metric look like in N hours?”* +- **promanomaly** answers: *“Is this metric behaving abnormally right now?”* + +## At a Glance + +| Aspect | promforecast | promanomaly | +|---------------------------|---------------------------------------------------|------------------------------------------------------| +| Core question | “What will this be in N hours?” | “Is this abnormal right now?” | +| Approach | Long-lookback model + forward prediction | Rolling-window baseline scoring | +| Lookback | Days to weeks | Minutes to hours (stratified detectors use days+) | +| Detection latency | Minutes to hours | Seconds to minutes | +| Best for | Capacity, traffic, seasonal KPIs | Bursty, sparse, step-changing, fleet-relative | +| Handles step changes | Slowly (smears into next refit) | Immediately (BOCPD / CUSUM) | +| Catches “one bad replica” | No | Yes (Cohort detector) | +| Output metrics | `*_forecast{…}`, bands, ETAs | `anomaly_score{…}`, `anomaly_outside_threshold{…}` | +| Compute cost | Heavier per series | Lightweight | + +Both tools share the exact same `(id, group)` label contract, so you can join their outputs in PromQL with zero rewriting. + +## Decision Tree + +``` +Start here +│ +├── Is the question “what will this metric look like in N hours”? +│ ├── Yes → promforecast +│ └── No → continue +│ +├── Does the signal have clear hourly/daily seasonality AND you need multi-hour forecasts? +│ ├── Yes → promforecast +│ └── No → continue +│ +├── Does the signal step-change on deploys, scale events, or feature flags? +│ ├── Yes → promanomaly (immediate change-point detection) +│ └── No → continue +│ +├── Is the anomaly “the signal completely stopped reporting”? +│ ├── Yes → promanomaly (`expect:` or dynamic absence) +│ └── No → continue +│ +├── Is the anomaly “one member of a fleet looks different from the rest”? +│ ├── Yes → promanomaly (Cohort detector) +│ └── No → continue +│ +├── Is the signal sparse, bursty, or heavy-tailed (p99 latency, rare errors, queue depth)? +│ ├── Yes → promanomaly (MAD / Hampel / IQR) +│ └── No → continue +│ +└── Reached here? + └── Smooth seasonal capacity/traffic metric → **both tools** work. + promforecast gives the long-horizon view; add promanomaly for fast break detection. +``` + +## When to Run Both + +There is **no overlap** to deduplicate. They catch different failure modes on the same signal: + +- promforecast → slow capacity creep, forecast-band breaches +- promanomaly → sudden step changes, one-bad-replica cases, sparse-event spikes + +They feed independent `PrometheusRule` files into the same Alertmanager. Use `inhibition_rules` keyed on `(id, group)` to suppress duplicate pages when both fire on the same outage. + +**Worked example** (B2B API with 50 replicas): +- promforecast on total request rate → capacity planning + ETAs +- promanomaly on the same rate (MAD + ZScoreEWMA) → fast deploy detection +- promanomaly Cohort on per-replica rate → “one bad replica” alerts +- Recording rule joins both on `(id, group)` for a single “API health” rollup + +## When to Pick Only One + +- **promforecast only** + Clean seasonal capacity/traffic metrics where long-horizon forecasts matter and you don’t need instant step-change or per-instance detection. + +- **promanomaly only** + Bursty/sparse signals, frequent regime shifts, or fleets where the canonical anomaly is “one member is different.” + +For almost everything else: **run both**. + +## What Neither Tool Does + +- Alerting or routing (that’s Alertmanager) +- Dashboards or UI (that’s Grafana) +- Causal analysis (“did the deploy cause this?”) +- Log/trace anomaly detection +- Full multivariate covariance across unrelated signals + +All advanced patterns are handled with standard PromQL, recording rules, or Alertmanager silences — see [`patterns.md`](patterns.md). \ No newline at end of file diff --git a/examples/configs/change-points.yaml b/examples/configs/change-points.yaml index 6bc9447..9ebdde2 100644 --- a/examples/configs/change-points.yaml +++ b/examples/configs/change-points.yaml @@ -5,7 +5,7 @@ # ordinary outlier detection — the two classes are complementary, not # overlapping (see docs/change-points.md). -apiVersion: promanomaly.io/v1alpha1 +apiVersion: promanomaly.io/v1 datasource: url: http://victoriametrics:8428/ diff --git a/examples/configs/cohort.yaml b/examples/configs/cohort.yaml index cbac03b..299b921 100644 --- a/examples/configs/cohort.yaml +++ b/examples/configs/cohort.yaml @@ -9,7 +9,7 @@ # This example also enables OpenTelemetry tracing — turn it on by # uncommenting the `telemetry:` block and installing the `[otel]` extra. -apiVersion: promanomaly.io/v1alpha1 +apiVersion: promanomaly.io/v1 datasource: url: http://victoriametrics:8428/ diff --git a/examples/configs/discovery.yaml b/examples/configs/discovery.yaml index 14d5cd9..227ea3a 100644 --- a/examples/configs/discovery.yaml +++ b/examples/configs/discovery.yaml @@ -15,7 +15,7 @@ # for the pre-deploy CI gate that catches wildcard mistakes before # they hit production cardinality. -apiVersion: promanomaly.io/v1alpha1 +apiVersion: promanomaly.io/v1 datasource: url: http://victoriametrics:8428/ diff --git a/examples/configs/ha.yaml b/examples/configs/ha.yaml index 09a0377..82f44ae 100644 --- a/examples/configs/ha.yaml +++ b/examples/configs/ha.yaml @@ -16,7 +16,7 @@ # - Bring your own Redis (Bitnami, AWS ElastiCache, in-cluster, …); # the umbrella ``promanomaly-stack`` chart does not bundle one yet. -apiVersion: promanomaly.io/v1alpha1 +apiVersion: promanomaly.io/v1 datasource: url: http://victoriametrics:8428/ diff --git a/examples/configs/self-monitoring.yaml b/examples/configs/self-monitoring.yaml index 1ee6393..18c35af 100644 --- a/examples/configs/self-monitoring.yaml +++ b/examples/configs/self-monitoring.yaml @@ -2,7 +2,7 @@ # metrics. This is both a smoke test (any rolling failure is itself an # anomaly) and a real production safeguard against silent stalls. -apiVersion: promanomaly.io/v1alpha1 +apiVersion: promanomaly.io/v1 datasource: url: http://victoriametrics:8428/ diff --git a/examples/configs/weekly-baseline.yaml b/examples/configs/weekly-baseline.yaml index 72e61aa..ddbcf39 100644 --- a/examples/configs/weekly-baseline.yaml +++ b/examples/configs/weekly-baseline.yaml @@ -12,7 +12,7 @@ # if you're not sure which stratification fits your signal. See # docs/scaling-stratified.md for the TSDB-cost projections. -apiVersion: promanomaly.io/v1alpha1 +apiVersion: promanomaly.io/v1 datasource: url: http://victoriametrics:8428/ diff --git a/examples/production/README.md b/examples/production/README.md new file mode 100644 index 0000000..46ed4b2 --- /dev/null +++ b/examples/production/README.md @@ -0,0 +1,85 @@ +# Production reference deployment + +A worked example of running promanomaly in production: two replicas +behind a Lease-based leader election, a shared Redis snapshot and +query cache, NetworkPolicy ingress + egress, anti-affinity, the +bundled PrometheusRule, and four detector groups sized for ~10k +output series. + +This directory is meant to be **copied and tuned**, not consumed +as-is. The values reflect a generic mid-size cluster; bring your own +Prometheus, your own VictoriaMetrics, your own Redis, and your own +PromQL queries. + +## Files + +| File | Purpose | +|---|---| +| [`architecture.txt`](architecture.txt) | ASCII data-flow diagram of the deployed topology. | +| [`values.yaml`](values.yaml) | Helm values for `charts/promanomaly`. Includes the four detector groups inline so `helm template --values values.yaml` renders end-to-end with no external state. | +| [`networkpolicy-egress.yaml`](networkpolicy-egress.yaml) | Sibling `NetworkPolicy` constraining egress to the TSDB, Redis, and the Kubernetes apiserver only. The chart-shipped `NetworkPolicy` covers ingress; this one covers egress. | +| [`argocd/application.yaml`](argocd/application.yaml) | Argo CD `Application` with two-source chart+values pinning. | +| [`flux/helmrepository.yaml`](flux/helmrepository.yaml) | Flux `HelmRepository` resolving the chart. | +| [`flux/helmrelease.yaml`](flux/helmrelease.yaml) | Flux `HelmRelease` referencing `existingConfigMap` so the detector groups are managed separately from chart upgrades. | +| [`flux/configmap.yaml`](flux/configmap.yaml) | The standalone detector config the `HelmRelease` references. | + +## Detector groups + +Four groups, each demonstrating a different anomaly shape: + +- **`error_rates`** — voting ensemble of `MAD` + `ZScoreEWMA`, gated by + a 5 % relative-significance floor so noise on tiny ratios doesn't + page. Highest priority. +- **`request_counts`** — multi-window `MAD` (`short` 15m + `long` 1h) + plus `Hampel` for spike isolation. Sized for bursty traffic. +- **`queue_depths`** — `IQR` for distribution-tail anomalies on sparse + signals, plus `BOCPD` for regime shifts. +- **`node_cohort`** — fleet-relative `Cohort` detection on CPU and + memory across node-exporter targets. Catches the "one bad node out + of 100" case forecast-based detection misses. + +Total output series ≈ 10k assuming roughly 100 series per query and +the configured cohort sizes. Bump `safety.max_series_per_query` and +`safety.max_total_series` proportionally if your fanout is wider. + +## Picking a GitOps tool + +The two GitOps samples — Argo CD under `argocd/` and Flux under +`flux/` — are equivalent end-states. Pick one; **don't mix them**. The +two patterns differ in one meaningful way: + +- **Argo CD's** sample uses a multi-source `Application` so chart + version and values file are pinned independently in the same + manifest. +- **Flux's** sample splits the values further: the chart-deployment + knobs live in the `HelmRelease`, and the detector groups live in a + sibling `ConfigMap` referenced via `existingConfigMap`. This keeps + chart upgrades and detector tuning on independent PR cadences, + which scales better for teams where SREs own the chart and data + scientists own the detector groups. + +Both pin `promanomaly` chart `1.0.0`. Bump via PR. + +## Tuning checklist + +Before applying any of this to a real cluster, walk through: + +1. **Replace placeholder service names** — `victoria-metrics-single-server`, + `promanomaly-redis-master`, and the namespaces (`observability`, + `argocd`, `flux-system`) all need to match your cluster. +2. **Validate the egress NetworkPolicy CIDR** — the sample assumes + the apiserver lives somewhere in `10.0.0.0/8`. Confirm with + `kubectl get svc kubernetes -o yaml`. +3. **Run `promanomaly validate --probe --strict`** against your TSDB + with these queries before merging. The pre-deploy gate catches + empty-result queries and over-broad selectors before they hit + production cardinality. +4. **Tune `alert_thresholds.score` per SLO** — the bundled defaults + are conservative; verifying against `make backtest-inject` is the + recommended path. +5. **Pin the image tag** — `image.tag: v1.0.0` in `values.yaml`. Do + not float `latest`. + +See [`docs/operations.md`](../../docs/operations.md) for the full +operational story, including the failover timeline and the sizing +table. diff --git a/examples/production/architecture.txt b/examples/production/architecture.txt new file mode 100644 index 0000000..5a769f6 --- /dev/null +++ b/examples/production/architecture.txt @@ -0,0 +1,112 @@ +Production reference architecture for promanomaly +================================================== + +Two-replica HA deployment in the ``observability`` namespace, reading +recent windows from a long-term VictoriaMetrics, sharing snapshot and +query state through Redis, and exposed to Prometheus / Grafana / +Alertmanager via the bundled ServiceMonitor and PrometheusRule. + +Data flow +--------- + + ┌───────────────────────────────────────────────────────────────────┐ + │ application namespaces │ + │ │ + │ workloads ──▶ /metrics │ + │ │ │ + └──────────────────┼────────────────────────────────────────────────┘ + │ + │ scrape + ▼ + ┌───────────────────────────────────────────────────────────────────┐ + │ observability namespace │ + │ │ + │ Prometheus / kube-prometheus-stack │ + │ │ │ + │ │ remote_write │ + │ ▼ │ + │ VictoriaMetrics (long-term, analytics plane) │ + │ ▲ │ + │ │ PromQL read (rolling window) │ + │ │ │ + │ ┌────┴─────────┐ Lease ┌──────────────┐ │ + │ │ promanomaly │ ◀──────────────▶ │ Kubernetes │ │ + │ │ (leader) │ │ apiserver │ │ + │ │ │ └──────────────┘ │ + │ │ scheduler ──▶│ │ + │ │ detectors │ │ + │ │ │ │ snapshot + query cache │ + │ │ └──────────┼─────────────────▶ ┌──────────┐ │ + │ │ │ │ Redis │ │ + │ │ /metrics ◀───┼─────────────────▶ │ │ │ + │ └────▲─────────┘ └────▲─────┘ │ + │ │ │ │ + │ ┌────┴─────────┐ │ │ + │ │ promanomaly │ ◀── snapshot reads ────┘ │ + │ │ (follower) │ │ + │ │ │ │ + │ │ /metrics ◀───┼──── (served from cache) │ + │ └────▲─────────┘ │ + │ │ │ + │ │ scrape │ + │ │ │ + │ Prometheus ──▶ Grafana │ + │ │ │ + │ └─▶ Alertmanager ──▶ (PagerDuty / Slack / Opsgenie / …) │ + │ │ + └───────────────────────────────────────────────────────────────────┘ + + +Topology notes +-------------- + + - Both detector replicas race for the Lease + (``coordination.k8s.io/v1/Lease/promanomaly-leader``). Only the + elected leader runs the scheduler; the follower serves + ``/metrics`` from the Redis snapshot cache. Failover triggers an + immediate detection run on the new leader so the next scrape + sees fresh scores — see docs/operations.md for the worked + timeline. + + - The query cache (also Redis) deduplicates overlapping rolling + windows across replicas: the leader writes, both leader and + follower can read. The ``key_prefix`` is shared with the + snapshot cache; they coexist in a single Redis database. + + - Pod anti-affinity (``kubernetes.io/hostname``) keeps the two + replicas on different nodes so a single node drain never takes + both. Set ``topology.kubernetes.io/zone`` instead on + multi-zone clusters where intra-zone scheduling is acceptable. + + - NetworkPolicy ingress (rendered by the chart) restricts + ``/-/reload`` and ``/debug/inspect`` to the observability + namespace. The sibling egress NetworkPolicy + (``examples/production/networkpolicy-egress.yaml``) constrains + egress to VictoriaMetrics, Redis, and the Kubernetes apiserver + only. + + - The bundled PrometheusRule fires the canonical alerts + (``AnomalyOutsideThreshold``, ``AnomalyLongRunning``, + ``AnomalyStale``, ``AnomalySourceFailing``, …); see + ``examples/alerts/promanomaly-rules.yaml`` for the full set + and docs/operations.md for the tuning matrix. + + +GitOps wiring +------------- + +Two equivalent end-states ship in this directory: + + argocd/ + application.yaml Argo CD ``Application`` with two-source + chart+values pinning. + + flux/ + helmrepository.yaml ``HelmRepository`` resolving the chart. + helmrelease.yaml ``HelmRelease`` with ``existingConfigMap`` + pointing at the sibling ConfigMap. + configmap.yaml Detector groups managed independently of + chart upgrades. + +Pick one; do not mix. Both pin ``promanomaly`` chart ``1.0.0``; +bump via PR. diff --git a/examples/production/argocd/application.yaml b/examples/production/argocd/application.yaml new file mode 100644 index 0000000..cd5500c --- /dev/null +++ b/examples/production/argocd/application.yaml @@ -0,0 +1,80 @@ +# Argo CD Application for the promanomaly detector chart. +# +# Pulls the chart from the project's published Helm repository and +# injects the ``values.yaml`` from ``examples/production/values.yaml`` +# in this repo via a single-source ``valueFiles`` reference. +# +# Usage: +# 1. Fork or clone the promanomaly repo; commit your tuned +# ``examples/production/values.yaml`` to a path your ArgoCD repo +# server can reach (or vendor it in your own GitOps repo). +# 2. Update ``repoURL`` / ``path`` below to point at that copy. +# 3. ``kubectl apply -f application.yaml`` against the ArgoCD +# namespace (typically ``argocd``). +# +# The chart-pinning happens via ``targetRevision``; bump it via PR. +# Auto-sync is off by default so the first install is operator-driven; +# flip ``automated:`` on once the deployment is shaped. + +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: promanomaly + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io + labels: + app.kubernetes.io/name: promanomaly + app.kubernetes.io/part-of: observability +spec: + project: default + + # Multi-source: chart from the OCI/Helm repo, values from the GitOps + # repo. ArgoCD merges them at render time. This keeps the chart + # version pinned in one place (this manifest) and the values pinned + # in another (the repo path) without copy-pasting either. + # + # ``repoURL`` mirrors the URL published in the top-level README + # quickstart and is rewritten by ``chart-releaser-action`` on tag. + # If you re-host the chart (private mirror, OCI registry, etc.), + # update both this manifest and your repo's quickstart so they + # don't drift. + sources: + - repoURL: https://esops-dev.github.io/promanomaly + chart: promanomaly + targetRevision: 1.0.0 + helm: + releaseName: promanomaly + valueFiles: + - $values/examples/production/values.yaml + - repoURL: https://github.com/esops-dev/promanomaly.git + targetRevision: v1.0.0 + ref: values + + destination: + server: https://kubernetes.default.svc + namespace: observability + + syncPolicy: + # Start operator-driven. Flip to ``automated`` once the rollout + # is comfortable; ArgoCD's diff view should be clean before that. + syncOptions: + - CreateNamespace=true + - ServerSideApply=true + - ApplyOutOfSyncOnly=true + retry: + limit: 5 + backoff: + duration: 10s + maxDuration: 3m + factor: 2 + + # IgnoreDifferences: the detector reconciles its Lease at runtime, + # which means ``coordination.k8s.io/v1/Lease`` is going to drift from + # whatever ArgoCD would otherwise render. Don't render or compare it. + ignoreDifferences: + - group: coordination.k8s.io + kind: Lease + jsonPointers: + - /spec + - /metadata/managedFields diff --git a/examples/production/flux/configmap.yaml b/examples/production/flux/configmap.yaml new file mode 100644 index 0000000..6ddd237 --- /dev/null +++ b/examples/production/flux/configmap.yaml @@ -0,0 +1,159 @@ +# Sibling ConfigMap holding the production detector configuration. +# +# Referenced by ``helmrelease.yaml`` via ``existingConfigMap`` so the +# detector's groups are managed independently of the chart upgrade +# cadence. This is the production-shaped pattern documented in +# docs/operations.md: chart bumps for the deployment surface, ConfigMap +# bumps for the detector groups, no cross-PR coupling. +# +# Keep the apiVersion at ``promanomaly.io/v1``; the alpha alias is +# documented in docs/config-schema.md and will be removed at v3.0. + +apiVersion: v1 +kind: ConfigMap +metadata: + name: promanomaly-detector-config + namespace: observability + labels: + app.kubernetes.io/name: promanomaly + app.kubernetes.io/managed-by: gitops +data: + config.yaml: | + apiVersion: promanomaly.io/v1 + + datasource: + url: "http://victoria-metrics-single-server.observability:8428/" + timeout: 10s + auth: + type: none + + server: + listen: ":9092" + refresh_interval: 1m + reload: + enabled: true + watch_configmap: true + auth: + type: none + + safety: + max_series_per_query: 1000 + max_total_series: 20000 + series_overflow: drop_lowest_priority + detect_timeout: 5s + query_timeout: 10s + on_source_failure: serve_stale + fail_ready_after: 3 + query_cache: + enabled: true + backend: redis + max_entries: 4096 + ttl: 30s + redis: + url: "redis://promanomaly-redis-master.observability:6379/0" + key_prefix: promanomaly + timeout: 2s + discovery: + forget_after_runs: 1000 + + highAvailability: + enabled: true + lease_name: promanomaly-leader + lease_namespace: observability + lease_duration: 15s + renew_deadline: 10s + retry_period: 2s + snapshot_ttl: 5m + + defaults: + window: 1h + step: 15s + min_points: 60 + warmup_policy: emit_warming_up + emit_baseline: true + emit_change_points: true + emit_baseline_stability: true + emit_duration: true + alert_thresholds: + score: 3.0 + + groups: + - name: error_rates + priority: 10 + ensemble: + method: voting + min_detectors: 2 + queries: + - id: api_error_ratio + promql: | + sum(rate(http_requests_total{job="api", code=~"5.."}[5m])) + / + sum(rate(http_requests_total{job="api"}[5m])) + min_relative_delta: 0.05 + detectors: + - name: MAD + - name: ZScoreEWMA + params: + alpha: 0.05 + alert_thresholds: + score: 4.0 + - id: ingest_error_ratio + promql: | + sum(rate(ingest_failures_total[5m])) + / + sum(rate(ingest_attempts_total[5m])) + min_relative_delta: 0.05 + detectors: + - name: MAD + - name: Hampel + params: + t0: 5 + + - name: request_counts + priority: 5 + queries: + - id: api_qps + promql: sum(rate(http_requests_total{job="api"}[1m])) + detectors: + - name: MAD + instance: short + params: + window: 15m + - name: MAD + instance: long + params: + window: 1h + - name: Hampel + params: + t0: 5 + + - name: queue_depths + priority: 5 + queries: + - id: worker_queue_depth + promql: sum by (queue) (queue_depth) + detectors: + - name: IQR + params: + k: 1.5 + - name: BOCPD + params: + hazard: 0.01 + + - name: node_cohort + priority: 3 + queries: + - id: cpu_busy_ratio + promql: 1 - rate(node_cpu_seconds_total{mode="idle"}[5m]) + detectors: + - name: Cohort + params: + cohort_label: instance + min_cohort_size: 5 + - id: memory_available_bytes + promql: node_memory_MemAvailable_bytes + detectors: + - name: Cohort + params: + cohort_label: instance + min_cohort_size: 5 diff --git a/examples/production/flux/helmrelease.yaml b/examples/production/flux/helmrelease.yaml new file mode 100644 index 0000000..afc1a46 --- /dev/null +++ b/examples/production/flux/helmrelease.yaml @@ -0,0 +1,54 @@ +# Flux HelmRelease for the promanomaly detector. +# +# Pairs with: +# - ``helmrepository.yaml`` (the chart source) +# - ``configmap.yaml`` (sibling ConfigMap referenced via +# ``existingConfigMap`` so detector groups +# are managed independently of chart upgrades) +# +# Pinning policy: ``spec.chart.spec.version`` is the only place the +# chart version is named. Bump via PR. The ``valuesFrom`` reference +# pulls the production values file from a ConfigMap sourced from this +# repo so the values are diffable in PRs. + +apiVersion: helm.toolkit.fluxcd.io/v2 +kind: HelmRelease +metadata: + name: promanomaly + namespace: observability +spec: + releaseName: promanomaly + targetNamespace: observability + + interval: 5m + timeout: 5m + + chart: + spec: + chart: promanomaly + version: 1.0.0 + sourceRef: + kind: HelmRepository + name: promanomaly + namespace: flux-system + + install: + createNamespace: true + remediation: + retries: 3 + upgrade: + cleanupOnFail: true + remediation: + retries: 3 + strategy: rollback + + # Detector groups managed out-of-band via the sibling ConfigMap. + # The chart's own ConfigMap rendering is skipped because + # ``existingConfigMap`` is set on the values block below. + valuesFrom: + - kind: ConfigMap + name: promanomaly-values + valuesKey: values.yaml + + values: + existingConfigMap: promanomaly-detector-config diff --git a/examples/production/flux/helmrepository.yaml b/examples/production/flux/helmrepository.yaml new file mode 100644 index 0000000..6b8dc66 --- /dev/null +++ b/examples/production/flux/helmrepository.yaml @@ -0,0 +1,16 @@ +# Flux HelmRepository pointing at the promanomaly chart repository. +# +# Reconciled every five minutes; bump ``interval`` down on clusters +# that need fast rollouts after a chart release (note that +# ``HelmRelease.upgrade.retries`` and ``install.remediation`` carry +# more weight for actually getting a new chart picked up than the +# refresh interval). + +apiVersion: source.toolkit.fluxcd.io/v1 +kind: HelmRepository +metadata: + name: promanomaly + namespace: flux-system +spec: + interval: 5m + url: https://esops-dev.github.io/promanomaly diff --git a/examples/production/networkpolicy-egress.yaml b/examples/production/networkpolicy-egress.yaml new file mode 100644 index 0000000..6142963 --- /dev/null +++ b/examples/production/networkpolicy-egress.yaml @@ -0,0 +1,81 @@ +# Explicit egress NetworkPolicy for promanomaly in production. +# +# The shipped chart NetworkPolicy only models ingress; this sibling +# manifest constrains egress to the three things the detector actually +# needs: the TSDB, Redis, and the Kubernetes API. Apply it after the +# chart so the chart-rendered NetworkPolicy (ingress) and this one +# (egress) coexist without one replacing the other. +# +# Adjust before applying: +# - ``spec.podSelector.matchLabels`` carries +# ``app.kubernetes.io/instance: promanomaly``. The chart sets this +# to ``.Release.Name``, so if you install with a release name other +# than ``promanomaly`` (e.g. ``helm install obs-detector ...``) +# the selector misses every pod and egress falls through to the +# cluster default. Mirror your release name here. +# - ``metadata.namespace`` (``observability``) must match the chart +# release namespace. +# - The TSDB destination (single-node VictoriaMetrics, vmselect, Mimir). +# - The Kubernetes API endpoint if your cluster uses a non-default +# apiserver service (rare; the 10.0.0.1 / kubernetes.default approach +# below works in most clusters). +# +# kubeconform-friendly: 0 CRDs, only built-in resources. + +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: promanomaly-egress + namespace: observability + labels: + app.kubernetes.io/name: promanomaly + app.kubernetes.io/managed-by: gitops +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: promanomaly + app.kubernetes.io/instance: promanomaly + policyTypes: + - Egress + egress: + # VictoriaMetrics single-node (or vmselect on a cluster install). + - to: + - podSelector: + matchLabels: + app.kubernetes.io/name: victoria-metrics-single + ports: + - port: 8428 + protocol: TCP + # Redis snapshot + query cache. + - to: + - podSelector: + matchLabels: + app.kubernetes.io/name: redis + ports: + - port: 6379 + protocol: TCP + # Kubernetes API — Lease coordination needs to reach the apiserver + # from every pod, not just the leader. The CIDR is the conventional + # in-cluster ClusterIP for the apiserver Service; replace with the + # actual range your cluster uses if it isn't 10.0.0.0/8. + - to: + - ipBlock: + cidr: 10.0.0.0/8 + ports: + - port: 443 + protocol: TCP + - port: 6443 + protocol: TCP + # DNS — required so the in-cluster service names above resolve. + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + podSelector: + matchLabels: + k8s-app: kube-dns + ports: + - port: 53 + protocol: UDP + - port: 53 + protocol: TCP diff --git a/examples/production/values.yaml b/examples/production/values.yaml new file mode 100644 index 0000000..dc40205 --- /dev/null +++ b/examples/production/values.yaml @@ -0,0 +1,314 @@ +# Production-shaped Helm values for promanomaly. +# +# This file is the operator-facing summary: the detector configuration +# lives below under ``groups:`` and the operational knobs (replicas, +# anti-affinity, NetworkPolicy egress) are tuned for a real cluster +# with a sibling Redis and a long-term TSDB on a separate namespace. +# +# Sized for ~10k output series across four groups (error rates, +# request counts, queue depths, and a per-node cohort fleet). Adjust +# ``max_series_per_query`` / ``max_total_series`` if you fan out +# wider. See examples/production/architecture.txt for the data flow. +# +# Image bump policy: pin a specific tag (do not float ``latest``). The +# same applies to the VM and Redis charts referenced by the ArgoCD / +# Flux samples in this directory. + +image: + repository: ghcr.io/esops-dev/promanomaly + # Pin to a specific app version. Bump in lock-step with the chart's + # ``appVersion`` (PR review surface) — leaving this empty falls back + # to ``Chart.appVersion``, which is fine if you trust chart upgrades + # to be the source of truth for which detector image runs. Floating + # ``latest`` is not supported. + tag: v1.0.0 + pullPolicy: IfNotPresent + +replicaCount: 2 # leader + one follower; bump to 3 for + # stronger availability during rolling + # restarts of large clusters. + +strategy: + # Auto-flipped to RollingUpdate by the chart when HA is on, but + # set here too so a ``helm get values`` reflects intent. + type: RollingUpdate + +# Pod anti-affinity: spread replicas across nodes so a single node +# drain never takes both pods at once. A topologyKey of +# ``kubernetes.io/hostname`` is the safe default; switch to +# ``topology.kubernetes.io/zone`` on clusters where intra-zone +# scheduling is acceptable. +affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/name: promanomaly + app.kubernetes.io/instance: promanomaly + topologyKey: kubernetes.io/hostname + +datasource: + # Long-term store. The detector reads recent windows only; production + # operators commonly point at a VictoriaMetrics single-node or + # Mimir/Thanos query frontend living in the observability namespace. + url: http://victoria-metrics-single-server.observability:8428/ + timeout: 10s + auth: + type: none + existingSecret: "" + +server: + listen: ":9092" + refresh_interval: 1m + reload: + enabled: true + watch_configmap: true + auth: + type: none # NetworkPolicy is the primary control + existingSecret: "" + +safety: + max_series_per_query: 1000 + max_total_series: 20000 + series_overflow: drop_lowest_priority + detect_timeout: 5s + query_timeout: 10s + on_source_failure: serve_stale + fail_ready_after: 3 + query_cache: + enabled: true + backend: redis # shared across replicas in HA + max_entries: 4096 + ttl: 30s + redis: + url: redis://promanomaly-redis-master.observability:6379/0 + key_prefix: promanomaly + timeout: 2s + discovery: + forget_after_runs: 1000 + +highAvailability: + enabled: true + lease_name: promanomaly-leader + lease_namespace: "" # release namespace at template time + lease_duration: 15s + renew_deadline: 10s + retry_period: 2s + snapshot_ttl: 5m + +defaults: + window: 1h + step: 15s + min_points: 60 + warmup_policy: emit_warming_up + emit_baseline: true + emit_change_points: true + emit_baseline_stability: true + emit_duration: true + alert_thresholds: + score: 3.0 + min_abs_delta: 0.0 + min_relative_delta: 0.0 + +# Output-label filtering — production clusters typically carry a +# ``customer`` or ``tenant`` label across service metrics. Drop those +# at the exporter boundary unless your alerting really needs them. +exporter: + output_labels: + allow: [] + drop: [] + +# OpenTelemetry tracing — uncomment to wire up. The chart only renders +# the env vars; the detector image must be the ``[otel]`` extra build. +telemetry: + otlp: + endpoint: "" + insecure: true + service_name: promanomaly + +# NetworkPolicy: lock down ingress to /-/reload and debug endpoints, +# and constrain egress to the TSDB, Redis, and Kubernetes API only. +# The shipped chart template only models ingress; the additional egress +# rules below ship as a sibling NetworkPolicy in +# examples/production/networkpolicy-egress.yaml so they can be +# managed/turned-off independently. +networkPolicy: + enabled: true + allowedScrapeNamespaces: + - observability # where Prometheus / kube-prometheus lives + +serviceMonitor: + enabled: true + interval: 30s + scrapeTimeout: 10s + labels: + release: prometheus # match the kube-prometheus-stack default + honorLabels: true + +# Bundled PrometheusRule. +# +# The chart ships this off by default because we don't carry opinionated +# thresholds. The reference deployment flips it on as a sensible +# starting point: every threshold below is a **default to be tuned per +# SLO**, not a recommendation. Review them against your own paging +# tolerance before merging — in particular ``durationSeconds`` on the +# long-running rule and ``maxStaleSeconds`` on the staleness rule are +# the two operators most often need to relax. +prometheusRule: + enabled: true + labels: + release: prometheus + rules: + anomalyOutsideThreshold: + enabled: true + for: 5m + severity: warning + anomalyLongRunning: + enabled: true + durationSeconds: 1800 + for: 1m + severity: critical + anomalyEnsembleAgreement: + enabled: true + for: 5m + severity: warning + anomalyChangePoint: + enabled: true + window: 10m + severity: info + anomalySignalMissing: + enabled: true + for: 10m + severity: critical + anomalyConfidenceLow: + enabled: true + threshold: 0.5 + for: 10m + severity: warning + anomalyStale: + enabled: true + maxStaleSeconds: 600 + for: 5m + severity: warning + anomalyDetectorFailing: + enabled: true + rateThreshold: 0.5 + window: 15m + for: 5m + severity: warning + anomalySourceFailing: + enabled: true + window: 15m + for: 10m + severity: warning + anomalyNotReady: + enabled: true + for: 5m + severity: critical + +resources: + requests: + cpu: 250m + memory: 512Mi + limits: + cpu: "1" + memory: 1Gi + +# Multi-group detector config. Production operators typically keep the +# detector groups in a sibling ConfigMap managed by GitOps (see +# ``existingConfigMap`` and the ArgoCD/Flux samples). The inline form +# below is kept self-contained so ``helm template --values values.yaml`` +# renders end-to-end without external state. +groups: + # Error rates — sustained anomalies on these tend to map directly to + # user-facing incidents. Ensemble voting reduces single-detector + # noise without flap suppression in the detector itself; the + # remaining flap concern goes to the ``for:`` clauses on the + # bundled PrometheusRules. + - name: error_rates + priority: 10 + ensemble: + method: voting + min_detectors: 2 + queries: + - id: api_error_ratio + promql: | + sum(rate(http_requests_total{job="api", code=~"5.."}[5m])) + / + sum(rate(http_requests_total{job="api"}[5m])) + min_relative_delta: 0.05 + detectors: + - name: MAD + - name: ZScoreEWMA + params: + alpha: 0.05 + alert_thresholds: + score: 4.0 + - id: ingest_error_ratio + promql: | + sum(rate(ingest_failures_total[5m])) + / + sum(rate(ingest_attempts_total[5m])) + min_relative_delta: 0.05 + detectors: + - name: MAD + - name: Hampel + params: + t0: 5 + + # Request counts — bursty by design, so Hampel handles single-point + # spikes while a second MAD instance with a longer window keeps the + # baseline from absorbing a clustered anomaly. + - name: request_counts + priority: 5 + queries: + - id: api_qps + promql: sum(rate(http_requests_total{job="api"}[1m])) + detectors: + - name: MAD + instance: short + params: + window: 15m + - name: MAD + instance: long + params: + window: 1h + - name: Hampel + params: + t0: 5 + + # Queue depths — typically sparse and non-Gaussian. IQR + change-point + # detection catches both magnitude and regime shifts. + - name: queue_depths + priority: 5 + queries: + - id: worker_queue_depth + promql: sum by (queue) (queue_depth) + detectors: + - name: IQR + params: + k: 1.5 + - name: BOCPD + params: + hazard: 0.01 + + # Per-node cohort comparison — flag the one bad node in a fleet of + # nominally identical workers. Sized for ~100 nodes; bump + # ``min_cohort_size`` if the cluster is larger. + - name: node_cohort + priority: 3 + queries: + - id: cpu_busy_ratio + promql: 1 - rate(node_cpu_seconds_total{mode="idle"}[5m]) + detectors: + - name: Cohort + params: + cohort_label: instance + min_cohort_size: 5 + - id: memory_available_bytes + promql: node_memory_MemAvailable_bytes + detectors: + - name: Cohort + params: + cohort_label: instance + min_cohort_size: 5