diff --git a/charts/promanomaly-metrics-adapter/README.md b/charts/promanomaly-metrics-adapter/README.md index 21a7099..62133cb 100644 --- a/charts/promanomaly-metrics-adapter/README.md +++ b/charts/promanomaly-metrics-adapter/README.md @@ -1,50 +1,35 @@ -# promanomaly-metrics-adapter +# promanomaly Metrics Adapter -An **opt-in** Kubernetes external/custom metrics adapter for promanomaly. +An **opt-in**, stateless Kubernetes external/custom metrics adapter for promanomaly. -It is a stateless aggregated apiserver that re-serves the anomaly metrics -the detector already wrote to the TSDB through -`external.metrics.k8s.io` / `custom.metrics.k8s.io`, so your existing HPA -and KEDA tooling can consume anomaly signal like any other metric. +It re-exposes the anomaly metrics written by the detector to the TSDB through the standard Kubernetes `external.metrics.k8s.io` and `custom.metrics.k8s.io` APIs. This lets HPA and KEDA scale on anomaly signals exactly like any other metric. -promanomaly stays the metrics **provider**; the autoscaler stays the -**decision-maker**. The adapter never makes a reaction decision itself. +promanomaly is the **metrics provider**. The autoscaler remains the **decision maker**. -## Guard-rails (read first) +## Guard-rails (Read First) -Anomaly signal is a *sharp* scaling input. Wire it carefully: +Anomaly signals are sharp scaling inputs. Use them carefully: -- **Scale on `anomaly_density`**, not raw `anomaly_score`. "Add workers - when the backlog is anomalously deep" is sound; "scale on how - anomalous a latency metric looks" *amplifies* incidents — a latency - spike scales you up, which can deepen the spike. -- **Gate on `anomaly_confidence_score` and `anomaly_duration_seconds`.** - React to sustained, high-confidence anomalies, not single-tick blips. -- **Always pair an anomaly-driven scaler with a reactive fallback HPA** - (CPU/memory or queue length). If the detector or TSDB is unavailable, - the fallback keeps the workload safe. +- **Scale on `anomaly_density`** (fraction of a group that is anomalous), not raw `anomaly_score`. +- **Gate on `anomaly_severity`** (already includes confidence and duration). +- **Always pair** an anomaly-driven scaler with a traditional reactive HPA fallback (CPU, memory, or queue depth). -The default `metrics` list deliberately omits scaling presets for -`anomaly_score`. Worked recipes with these guard-rails baked in live in -[`examples/k8s/`](../../examples/k8s/). +Safe configuration examples are in [`examples/k8s/`](../../examples/k8s/). -## What it exposes +## Exposed Metrics -| Metric | Good for | -| --- | --- | -| `anomaly_density` | Fraction of a group currently anomalous — the recommended scaling input. | -| `anomaly_severity` | Operator-facing 0-1 severity; gate scaling on it. | -| `anomaly_active_series` | Count of firing series in a group. | -| `anomaly_outside_threshold` | Per-series firing flag (0/1). | +| Metric | Recommended Use | +|-------------------------------|----------------------------------------------| +| `anomaly_density` | Primary scaling signal (fraction anomalous) | +| `anomaly_severity` | 0–1 normalized severity for gating | +| `anomaly_active_series` | Count of currently firing series | +| `anomaly_outside_threshold` | Per-series 0/1 firing flag | -Both API groups are served: +Both API groups are supported: +- `external.metrics.k8s.io` — recommended for KEDA and HPA `External` metrics. +- `custom.metrics.k8s.io` — for HPA `Object` and `Pods` metrics. -- **`external.metrics.k8s.io`** — cluster-scoped, the path KEDA's - `external` trigger and HPA `External` metric source use. Recommended. -- **`custom.metrics.k8s.io`** — the same signals associated with objects - (pods, namespaces) for HPA `Object`/`Pods` rules. - -## Install +## Quick Install ```bash helm install promanomaly-adapter charts/promanomaly-metrics-adapter \ @@ -52,45 +37,37 @@ helm install promanomaly-adapter charts/promanomaly-metrics-adapter \ --set datasource.url=http://victoriametrics.monitoring.svc:8428/ ``` -Verify: +Verify it is registered: ```bash kubectl get apiservices | grep metrics.k8s.io + kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq . ``` ## TLS -The aggregation layer only talks HTTPS to an APIService backend, so a -serving cert is mandatory. - -- **Default**: the chart mints a self-signed CA + serving cert and wires - the APIService `caBundle` to it. Simple, but it **regenerates on every - `helm upgrade`** (a brief reconcile blip). -- **Rotation-free**: set `tls.existingSecret` to a `kubernetes.io/tls` - Secret (e.g. from cert-manager) and `tls.caBundle` to its CA. No churn. -- **Skip verification**: `apiService.insecureSkipTLSVerify: true` if you - accept the aggregation layer not verifying the adapter cert. - -## Key values - -| Key | Default | Description | -| --- | --- | --- | -| `datasource.url` | `http://victoriametrics:8428/` | TSDB the detector writes to. | -| `datasource.auth.type` | `none` | `none`/`bearer`/`basic`/`mtls`. | -| `datasource.auth.existingSecret` | `""` | Secret with the credential (keys: `token`/`password`/`ca.crt`+`tls.crt`+`tls.key`), mounted and referenced by file. | -| `metrics` | the four above | Anomaly metrics to expose. | -| `customResources` | `[pods, namespaces]` | Object kinds for custom metrics. | -| `tls.existingSecret` | `""` | Use a managed serving cert instead of generating one. | -| `apiService.enabled` | `true` | Register the two APIService objects. | -| `apiService.insecureSkipTLSVerify` | `false` | Skip aggregation-layer cert verification. | -| `rbac.create` | `true` | auth-delegator + auth-reader + HPA metrics-reader RBAC. | -| `networkPolicy.enabled` | `true` | Restrict ingress to the serving port. | - -## How it differs from k8s-prometheus-adapter - -This adapter is purpose-built for the bounded set of anomaly metrics -promanomaly emits, with the guard-rails documented above. It reads the -same TSDB the detector writes to and translates label selectors directly -into PromQL matchers. For arbitrary Prometheus metrics, use the -general-purpose [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter). +The Kubernetes aggregation layer requires HTTPS. + +The chart can: +- Automatically generate a self-signed serving certificate (default, simple). +- Use an existing `kubernetes.io/tls` Secret via `tls.existingSecret` (recommended for production, e.g. from cert-manager). + +## Key Configuration Values + +| Key | Default | Description | +|-------------------------------------|--------------------------|-------------| +| `datasource.url` | — | URL of the TSDB written by the detector | +| `datasource.auth.type` | `none` | `none` / `bearer` / `basic` / `mtls` | +| `datasource.auth.existingSecret` | `""` | Secret containing credentials | +| `metrics` | Core anomaly metrics | Metrics to expose | +| `customResources` | `[pods, namespaces]` | Kubernetes objects for custom metrics | +| `tls.existingSecret` | `""` | Use a managed TLS Secret | +| `apiService.insecureSkipTLSVerify` | `false` | Skip TLS verification (not recommended in production) | +| `networkPolicy.enabled` | `true` | Restrict ingress | + +## How It Differs from prometheus-adapter + +This adapter is purpose-built and opinionated specifically for promanomaly’s anomaly metrics. It includes built-in guard-rails and directly translates label selectors into PromQL. + +For exposing arbitrary Prometheus metrics, use the general-purpose [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter) instead. \ No newline at end of file diff --git a/charts/promanomaly-stack/README.md b/charts/promanomaly-stack/README.md index 65d023d..42652e6 100644 --- a/charts/promanomaly-stack/README.md +++ b/charts/promanomaly-stack/README.md @@ -1,22 +1,29 @@ -# promanomaly-stack umbrella chart +# promanomaly-stack Umbrella Chart -Greenfield bundle: installs the [promanomaly](../promanomaly/README.md) -detector plus a VictoriaMetrics single-node TSDB in one ``helm -install``. Production users running an existing TSDB should install -the ``promanomaly`` chart on its own and point its ``datasource.url`` -at their long-term TSDB instead. +Greenfield bundle that installs the **promanomaly detector** + a **VictoriaMetrics single-node TSDB** in one `helm install`. + +> **Production note**: If you already have a VictoriaMetrics (or any Prometheus-compatible TSDB), install the plain [`promanomaly` chart](../promanomaly/README.md) instead and point `datasource.url` at your existing long-term store. + +## Quick Start ```bash helm repo add promanomaly https://esops-dev.github.io/promanomaly + helm dependency update charts/promanomaly-stack -helm install obs promanomaly/promanomaly-stack + +helm install obs promanomaly/promanomaly-stack \ + --namespace monitoring \ + --create-namespace ``` -Subchart values are nested under each subchart's name in -``values.yaml``: +## Configuration + +Subchart values are nested under each component: ```yaml promanomaly: + datasource: + url: http://{{ include "victoria-metrics-single.fullname" . }}:8428/ groups: - name: example queries: @@ -27,5 +34,10 @@ promanomaly: victoria-metrics-single: server: - retentionPeriod: 30 + retentionPeriod: 30d # adjust as needed + # ... other VM settings ``` + +Full reference is in `charts/promanomaly-stack/values.yaml`. + +That’s it — one command gives you a working detector + storage for immediate testing or small-scale production. For larger setups, switch to the standalone `promanomaly` chart with your own TSDB. \ No newline at end of file diff --git a/charts/promanomaly/README.md b/charts/promanomaly/README.md index 6501c27..0014204 100644 --- a/charts/promanomaly/README.md +++ b/charts/promanomaly/README.md @@ -1,60 +1,45 @@ -# promanomaly Helm chart +# promanomaly Helm Chart -Installs the promanomaly detector — a single ``Deployment``, a -``Service`` for ``/metrics``, a ``ConfigMap`` for the detector -configuration, an optional ``ServiceMonitor`` for Prometheus Operator -setups, an optional ``PrometheusRule`` carrying the reference alerts, -and a ``NetworkPolicy`` that locks down ``/-/reload`` and the debug -endpoints by default. +This chart installs the promanomaly detector as a single `Deployment`, with a `Service` for `/metrics`, a `ConfigMap` for your configuration, an optional `ServiceMonitor`, an optional `PrometheusRule` with reference alerts, and a `NetworkPolicy` that locks down `/-/reload` and debug endpoints by default. -Bring your own Prometheus/VictoriaMetrics. For a greenfield install -that bundles VictoriaMetrics, see the ``promanomaly-stack`` umbrella. +Bring your own Prometheus or VictoriaMetrics. For a full stack (including VictoriaMetrics), use the `promanomaly-stack` umbrella chart. -## Quick start +## Quick Start ```bash helm repo add promanomaly https://esops-dev.github.io/promanomaly + helm install my-detector promanomaly/promanomaly \ - --set datasource.url=http://my-victoriametrics:8428/ \ + --set datasource.url=http://victoriametrics:8428/ \ --values my-groups.yaml ``` -Where ``my-groups.yaml`` carries detector groups under ``groups:``. - -## Values - -See ``values.yaml`` and the schema in ``values.schema.json``. The -notable bits: - -- ``replicaCount`` defaults to ``1``. Single-replica is the simplest - topology; HA mode (``highAvailability.enabled: true``) is supported - and adds Lease-based leader election + a Redis-backed snapshot - cache so multiple replicas serve the same ``/metrics`` snapshot. - See ``examples/configs/ha.yaml`` and ``examples/production/``. -- ``existingConfigMap`` lets GitOps tooling (Argo CD, Flux) manage the - detector configuration out-of-band; the chart then skips its own - ConfigMap rendering. -- ``datasource.auth`` supports ``none``/``bearer``/``basic``/``mtls``. - For anything other than ``none``, set - ``datasource.auth.existingSecret`` to a Secret carrying the credential - — the chart mounts it as files and references it by path so it never - lands in the ConfigMap. Expected Secret keys: ``token`` (bearer), - ``password`` (basic; set ``datasource.auth.username`` too), or - ``ca.crt``/``tls.crt``/``tls.key`` (mTLS). -- ``networkPolicy.enabled`` defaults on; turn it off if your cluster - does not use NetworkPolicies. -- ``serviceMonitor.honorLabels: true`` is the default and required - for anomaly labels to flow through to Prometheus unchanged. -- ``prometheusRule.enabled`` is **off** by default — we don't ship - opinionated thresholds. The bundled rules mirror - ``examples/alerts/promanomaly-rules.yaml`` and expose - per-rule toggles + tunable thresholds so operators can adopt the - ones they want without templating from scratch. - -## Linting locally +`my-groups.yaml` contains your detector groups (under the `groups:` key). + +## Key Configuration Options + +| Value | Default | Description | +|------------------------------------|-------------|-------------| +| `replicaCount` | `1` | Single replica is simplest and fully supported | +| `highAvailability.enabled` | `false` | Enables leader election + Redis-backed snapshot cache for HA | +| `existingConfigMap` | — | Use your own ConfigMap (GitOps-friendly) | +| `datasource.auth` | `none` | Supports `none` / `bearer` / `basic` / `mtls` | +| `datasource.auth.existingSecret` | — | Secret for credentials (mounted as files) | +| `networkPolicy.enabled` | `true` | Locks down sensitive endpoints | +| `serviceMonitor.enabled` | `false` | Creates ServiceMonitor for Prometheus Operator | +| `serviceMonitor.honorLabels` | `true` | Required for correct label propagation | +| `prometheusRule.enabled` | `false` | Installs reference alerts (tunable thresholds) | + +Full reference in `values.yaml` + `values.schema.json`. + +## Local Linting ```bash helm dependency update charts/promanomaly + helm lint --strict charts/promanomaly + helm template charts/promanomaly | kubeconform -strict -summary -ignore-missing-schemas ``` + +That’s it. The chart is deliberately minimal and production-ready out of the box. See the [production-readiness checklist](../production-checklist.md) for the full go-live steps. \ No newline at end of file diff --git a/detector/src/promanomaly/detectors/bocpd.py b/detector/src/promanomaly/detectors/bocpd.py index 8bcfe6f..f322c52 100644 --- a/detector/src/promanomaly/detectors/bocpd.py +++ b/detector/src/promanomaly/detectors/bocpd.py @@ -256,13 +256,25 @@ def fit_score( # the cumulative mass at the recent run-lengths — the standard # way to surface the change-point signal from a BOCPD posterior. recent_lag = min(lag + 1, weights.size) - # Clamp into [0, 1] — the weights *should* be a normalised - # probability distribution, but the underlying floating-point - # logsumexp + scipy.special.gammaln path occasionally returns - # 1.0 + 1e-16 due to round-off. Clamping keeps the published - # score within the documented contract without changing the - # firing decision (the threshold is far from the edges). - cp_prob = float(min(1.0, max(0.0, weights[:recent_lag].sum()))) + # Insufficient-points guard. The run-length posterior holds one + # more state than the number of observed samples, so when the + # window has no more than ``lag`` samples (``weights.size <= + # lag + 1``) the "recent" window already spans every run length. + # The cumulative mass is then trivially ~1.0 and would fire a + # spurious change-point on any signal — there is no older tail to + # contrast the recent mass against. Report no change-point in that + # regime, mirroring the other detectors' zero-score behaviour on + # too-short input. (Normally masked by ``min_points`` >> ``lag``.) + if weights.size <= lag + 1: + cp_prob = 0.0 + else: + # Clamp into [0, 1] — the weights *should* be a normalised + # probability distribution, but the underlying floating-point + # logsumexp + scipy.special.gammaln path occasionally returns + # 1.0 + 1e-16 due to round-off. Clamping keeps the published + # score within the documented contract without changing the + # firing decision (the threshold is far from the edges). + cp_prob = float(min(1.0, max(0.0, weights[:recent_lag].sum()))) # MAP-style baseline: weighted average of regime means by their # posterior run-length mass. Falls back to the overall mean if # the posterior is degenerate (all zero mass). diff --git a/detector/src/promanomaly/detectors/distribution_shift.py b/detector/src/promanomaly/detectors/distribution_shift.py index 41e8225..03b2f8f 100644 --- a/detector/src/promanomaly/detectors/distribution_shift.py +++ b/detector/src/promanomaly/detectors/distribution_shift.py @@ -114,8 +114,11 @@ def fit_score( mad = float(np.median(np.abs(baseline - baseline_median))) score = dist / mad if mad > 0.0 else 0.0 elif statistic == "energy": - # Energy distance approximation via sorted samples. - dist = float(stats.wasserstein_distance(baseline, recent)) + # Energy distance: sensitive to differences across the whole + # distribution (not just the optimal-transport cost Wasserstein + # measures), normalised by the baseline MAD like the + # Wasserstein branch so the two share a score scale. + dist = float(stats.energy_distance(baseline, recent)) mad = float(np.median(np.abs(baseline - baseline_median))) score = dist / mad if mad > 0.0 else 0.0 else: diff --git a/detector/src/promanomaly/detectors/seasonal_hybrid_esd.py b/detector/src/promanomaly/detectors/seasonal_hybrid_esd.py index ca2b339..834831e 100644 --- a/detector/src/promanomaly/detectors/seasonal_hybrid_esd.py +++ b/detector/src/promanomaly/detectors/seasonal_hybrid_esd.py @@ -8,10 +8,16 @@ This is still baseline detection, not forecasting: it scores residuals against historical structure, it does not predict future values. -Score is the ratio of the ESD test statistic to the critical value — -values above 1.0 indicate the residual is an outlier at the configured -significance level. Normalised so the default -``alert_thresholds.score=3.0`` fires on clearly significant outliers. +Score is the ratio of the ESD test statistic to the critical value: a +score of ~1.0 means the latest residual sits exactly at the configured +``alpha`` significance boundary, and higher scores are progressively more +extreme outliers. Note this scale differs from the sigma-multiplier scale +of MAD/Hampel — the universal default ``alert_thresholds.score=3.0`` is +deliberately conservative here (it fires only well past the significance +boundary), so groups that want firing at the ESD significance level should +set a lower per-query/per-detector ``alert_thresholds.score`` (e.g. ~1.0). +The detector leaves ``is_outside`` to the runner's threshold check rather +than asserting its own significance verdict. """ from __future__ import annotations diff --git a/detector/src/promanomaly/exporter.py b/detector/src/promanomaly/exporter.py index 58bc761..c56edba 100644 --- a/detector/src/promanomaly/exporter.py +++ b/detector/src/promanomaly/exporter.py @@ -15,6 +15,7 @@ from __future__ import annotations +import math import re import time from collections import defaultdict @@ -97,6 +98,22 @@ def validate_label_name(name: str) -> str: "Hampel, cross-cohort median for Cohort)." ), ), + "anomaly_baseline_upper": ( + "gauge", + ( + "Upper edge of the detector's expected band, when the detector " + "models an explicit interval (e.g. STLResidualMAD). Omitted by " + "detectors that emit only a point baseline." + ), + ), + "anomaly_baseline_lower": ( + "gauge", + ( + "Lower edge of the detector's expected band, when the detector " + "models an explicit interval (e.g. STLResidualMAD). Omitted by " + "detectors that emit only a point baseline." + ), + ), "anomaly_warming_up": ( "gauge", "1 while the rolling window is still filling toward min_points.", @@ -222,12 +239,31 @@ def render_label_value(value: str) -> str: return value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n") +def render_value(value: float) -> str: + """Render a float in Prometheus exposition form. + + Non-finite values map to the canonical ``NaN`` / ``+Inf`` / ``-Inf`` + tokens the exposition format defines. Python's default float + stringification emits lowercase ``nan`` / ``inf`` / ``-inf``; Go's + ``ParseFloat`` (and VictoriaMetrics) accept those case-insensitively, + but the canonical spelling keeps ``/metrics`` portable across every + Prometheus-compatible parser rather than relying on that leniency. + Finite values stringify unchanged. + """ + if math.isfinite(value): + return repr(value) + if math.isnan(value): + return "NaN" + return "+Inf" if value > 0 else "-Inf" + + def _format_sample(sample: Sample) -> str: """One ``metric{labels} value`` line.""" + value = render_value(sample.value) if sample.labels: label_pairs = ",".join(f'{k}="{render_label_value(v)}"' for k, v in sample.labels) - return f"{sample.metric}{{{label_pairs}}} {sample.value}" - return f"{sample.metric} {sample.value}" + return f"{sample.metric}{{{label_pairs}}} {value}" + return f"{sample.metric} {value}" def render_samples(samples: Iterable[Sample]) -> str: diff --git a/detector/src/promanomaly/httpauth.py b/detector/src/promanomaly/httpauth.py index 5f65d8b..263e51b 100644 --- a/detector/src/promanomaly/httpauth.py +++ b/detector/src/promanomaly/httpauth.py @@ -132,7 +132,7 @@ def _do_init(self) -> None: ) from exc gcp_cfg = self._auth.gcp if gcp_cfg and gcp_cfg.credentials_file: - self._gcp_credentials, _ = google.auth.load_credentials_from_file( # type: ignore[no-untyped-call] + self._gcp_credentials, _ = google.auth.load_credentials_from_file( # type: ignore[no-untyped-call, unused-ignore] gcp_cfg.credentials_file, scopes=["https://www.googleapis.com/auth/monitoring.read"], ) diff --git a/detector/src/promanomaly/inspect.py b/detector/src/promanomaly/inspect.py index a2e24c9..b0517be 100644 --- a/detector/src/promanomaly/inspect.py +++ b/detector/src/promanomaly/inspect.py @@ -72,6 +72,13 @@ def parse_label_selector(value: str | None) -> dict[str, str]: Returns an empty dict for ``None`` or the empty string — meaning "match anything", consistent with how ``promtool query`` treats an empty matcher. + + Values may be bare (``endpoint=/api/users``) or quoted + (``endpoint="/api/users"``) — matching surrounding single or double + quotes are stripped. ``/metrics`` renders label values *with* quotes + and operators think in PromQL, so accepting the quoted form avoids a + silent zero-match when a selector is copy-pasted from a scrape or a + PromQL matcher. """ if not value: return {} @@ -87,13 +94,20 @@ def parse_label_selector(value: str | None) -> dict[str, str]: ) key, _, val = raw.partition("=") key = key.strip() - val = val.strip() + val = _strip_matching_quotes(val.strip()) if not key: raise InspectError("bad_selector", f"label selector entry {raw!r}: empty key") selector[key] = val return selector +def _strip_matching_quotes(val: str) -> str: + """Drop one layer of matching surrounding single or double quotes.""" + if len(val) >= 2 and val[0] == val[-1] and val[0] in ("'", '"'): + return val[1:-1] + return val + + def _resolve(config: Config, metric_id: str) -> _ResolvedQuery: """Find the (group, query) whose ``id`` matches ``metric_id``. diff --git a/detector/src/promanomaly/runner.py b/detector/src/promanomaly/runner.py index 83f0401..45be3e5 100644 --- a/detector/src/promanomaly/runner.py +++ b/detector/src/promanomaly/runner.py @@ -20,6 +20,7 @@ import asyncio import concurrent.futures import contextvars +import math import os import time from dataclasses import dataclass, field @@ -93,6 +94,7 @@ REASON_INSUFFICIENT_DATA = "insufficient_data" REASON_UNKNOWN_DETECTOR = "unknown_detector" REASON_QUERY_EXCEPTION = "query_exception" +REASON_SERIES_EXCEPTION = "series_exception" REASON_CARDINALITY = "cardinality_overflow" # Per-dimension cap on the fleet-density slice partition @@ -1103,14 +1105,36 @@ async def _process_query_variant( for series in scored_series: out.series_count += 1 - produced = await self._score_series( - group=group, - query=effective_query, - plans=plans, - series=series, - defaults=defaults, - cohort_context=cohort_context, - ) + # Per-series failure isolation: the per-detector loop inside + # _score_series guards the detector math, but the surrounding + # threshold/ensemble/label/format code runs unguarded. A single + # malformed series must not abort scoring of the remaining + # series in this query or the rest of the group's run (the + # documented "one failure doesn't cascade" contract). Catch + # Exception — not BaseException — so leader-demotion task + # cancellation still propagates. + try: + produced = await self._score_series( + group=group, + query=effective_query, + plans=plans, + series=series, + defaults=defaults, + cohort_context=cohort_context, + ) + except Exception as exc: + logger.exception( + "series_failed", + group=group.name, + query=effective_query.id, + error=str(exc), + ) + self._ops.failures_total.labels( + group=group.name, + detector="", + reason=REASON_SERIES_EXCEPTION, + ).inc() + continue out.samples.extend(produced) @property @@ -1633,6 +1657,18 @@ def _format_detector_samples( samples.append(Sample(metric="anomaly_best_detector", labels=labels, value=1.0)) if defaults.emit_baseline: samples.append(Sample(metric="anomaly_baseline", labels=labels, value=baseline)) + # Optional baseline band. Only detectors that model an explicit + # interval (e.g. STLResidualMAD) populate these columns; the + # rest omit them. Emit each only when present and finite so a + # detector that doesn't produce a band never publishes a + # misleading 0.0 upper/lower edge. + for column, band_metric in ( + ("baseline_upper", "anomaly_baseline_upper"), + ("baseline_lower", "anomaly_baseline_lower"), + ): + band = score_row.get(column) + if band is not None and math.isfinite(float(band)): + samples.append(Sample(metric=band_metric, labels=labels, value=float(band))) if defaults.emit_duration: samples.append( Sample( diff --git a/detector/tests/test_bocpd.py b/detector/tests/test_bocpd.py index eb5e163..11211a1 100644 --- a/detector/tests/test_bocpd.py +++ b/detector/tests/test_bocpd.py @@ -118,3 +118,34 @@ def test_change_point_column_present() -> None: df = clean_baseline(n=120) row = detector.fit_score(df, "", {}).iloc[-1] assert "change_point" in row.index + + +def test_short_window_does_not_saturate() -> None: + """A window no longer than ``lag`` must not fire a spurious change-point. + + The score is the posterior mass at run lengths <= lag; when the window + has <= lag samples every run length is "recent", so without a guard the + cumulative mass is trivially ~1.0 and fires on any signal. A clean + baseline that short must report no change-point. + """ + detector = BOCPD() + df = clean_baseline(n=12, mean=10.0, noise=0.1) # n <= default lag (15) + row = detector.fit_score(df, "", {}).iloc[-1] + assert float(row["score"]) == 0.0 + assert not bool(row["change_point"]) + assert not bool(row["is_outside"]) + # The baseline is still a usable estimate of the (single) regime mean. + assert float(row["baseline"]) == pytest.approx(10.0, abs=1.0) + + +def test_short_window_guard_respects_lag_param() -> None: + # With a smaller lag the same window is long enough to leave a tail, + # so the guard does not suppress scoring. + detector = BOCPD() + df = clean_baseline(n=12, mean=10.0, noise=0.1) + guarded = detector.fit_score(df, "", {"lag": 15}).iloc[-1] + scored = detector.fit_score(df, "", {"lag": 3}).iloc[-1] + assert float(guarded["score"]) == 0.0 + # lag=3 leaves run-length states beyond the recent window, so the + # detector computes a real (here still-low, clean-baseline) score. + assert float(scored["score"]) < 0.5 diff --git a/detector/tests/test_distribution_shift.py b/detector/tests/test_distribution_shift.py index e244ccc..2fffbbd 100644 --- a/detector/tests/test_distribution_shift.py +++ b/detector/tests/test_distribution_shift.py @@ -49,6 +49,27 @@ def test_wasserstein_statistic() -> None: assert float(row["score"]) > 0.0 +def test_energy_statistic() -> None: + detector = DistributionShift() + df = variance_shift(n=200, noise_low=0.1, noise_high=2.0, change_at=100) + row = detector.fit_score(df, "", {"statistic": "energy"}).iloc[-1] + assert float(row["score"]) > 0.0 + + +def test_energy_is_not_wasserstein() -> None: + # The "energy" statistic must compute the energy distance, not alias + # the Wasserstein branch. They share the MAD normalisation but are + # distinct two-sample distances, so their scores on a shifted window + # must differ (regression guard: the energy branch previously called + # wasserstein_distance and produced byte-identical scores). + detector = DistributionShift() + df = variance_shift(n=200, noise_low=0.1, noise_high=2.0, change_at=100) + energy = float(detector.fit_score(df, "", {"statistic": "energy"}).iloc[-1]["score"]) + wasserstein = float(detector.fit_score(df, "", {"statistic": "wasserstein"}).iloc[-1]["score"]) + assert energy > 0.0 + assert energy != pytest.approx(wasserstein) + + def test_invalid_statistic_raises() -> None: detector = DistributionShift() df = clean_baseline(n=100) diff --git a/detector/tests/test_exporter.py b/detector/tests/test_exporter.py index 7ee8fd7..8975b17 100644 --- a/detector/tests/test_exporter.py +++ b/detector/tests/test_exporter.py @@ -51,6 +51,23 @@ def test_label_escaping() -> None: assert render_label_value("line\nbreak") == "line\\nbreak" +def test_non_finite_values_use_canonical_tokens() -> None: + # The exposition format requires NaN / +Inf / -Inf (Python's default + # float stringification emits lowercase nan / inf / -inf). Render the + # canonical spelling so /metrics is portable across every parser. + samples = [ + Sample(metric="anomaly_score", labels=(("id", "n"),), value=float("nan")), + Sample(metric="anomaly_score", labels=(("id", "p"),), value=float("inf")), + Sample(metric="anomaly_score", labels=(("id", "m"),), value=float("-inf")), + ] + out = render_samples(samples) + assert 'anomaly_score{id="n"} NaN' in out + assert 'anomaly_score{id="p"} +Inf' in out + assert 'anomaly_score{id="m"} -Inf' in out + # No lowercase leakage. + assert " nan" not in out and " inf" not in out and " -inf" not in out + + def test_render_rejects_colon_in_metric() -> None: samples = [Sample(metric="bad:name", labels=(), value=1.0)] with pytest.raises(InvalidMetricNameError): diff --git a/detector/tests/test_inspect.py b/detector/tests/test_inspect.py index 29fe466..e435681 100644 --- a/detector/tests/test_inspect.py +++ b/detector/tests/test_inspect.py @@ -44,6 +44,18 @@ def test_parse_label_selector_handles_empty_and_keys() -> None: assert parse_label_selector("a=1, b=2") == {"a": "1", "b": "2"} +def test_parse_label_selector_strips_matching_quotes() -> None: + # /metrics renders label values with quotes and operators think in + # PromQL, so a copy-pasted quoted selector must match the same series + # as the bare form rather than silently matching nothing. + assert parse_label_selector('endpoint="/api/users"') == {"endpoint": "/api/users"} + assert parse_label_selector("endpoint='/api/users'") == {"endpoint": "/api/users"} + assert parse_label_selector('a="1", b=2') == {"a": "1", "b": "2"} + # Mismatched / non-enclosing quotes are left untouched. + assert parse_label_selector('a="1') == {"a": '"1'} + assert parse_label_selector('a=1"') == {"a": '1"'} + + def test_parse_label_selector_rejects_bare_key() -> None: with pytest.raises(InspectError) as exc: parse_label_selector("nope") diff --git a/detector/tests/test_runner.py b/detector/tests/test_runner.py index e3ab118..34744a7 100644 --- a/detector/tests/test_runner.py +++ b/detector/tests/test_runner.py @@ -14,6 +14,7 @@ from promanomaly.runner import ( REASON_DETECTOR_EXCEPTION, REASON_QUERY_EXCEPTION, + REASON_SERIES_EXCEPTION, REASON_TIMEOUT, REASON_UNKNOWN_DETECTOR, Runner, @@ -289,6 +290,113 @@ def fit_score(self, *_: Any, **__: Any) -> pd.DataFrame: registry.unregister("Slow") +@pytest.mark.asyncio +async def test_per_series_exception_does_not_block_other_series( + stub_source: StubSource, +) -> None: + # A failure in _score_series *outside* the guarded per-detector loop + # (threshold / ensemble / label / format code) must not abort scoring + # of the remaining series in the query — the "one failure doesn't + # cascade" contract. Simulate it by making _score_series raise for one + # series only. + good_a = clean_baseline(n=120) + good_b = point_spike(n=120) + stub_source.respond( + lambda _: StubResponse( + series=[ + make_series({"instance": "host-a"}, good_a), + make_series({"instance": "host-b"}, good_b), + ] + ) + ) + + cfg = _config(min_points=30) + runner, _store, ops = _runner(cfg, stub_source) + + original = runner._score_series + + async def flaky(*, series: Any, **kwargs: Any) -> Any: + if series.labels.get("instance") == "host-a": + raise RuntimeError("intentional series failure") + return await original(series=series, **kwargs) + + runner._score_series = flaky # type: ignore[method-assign] + + result = await runner.run_group("g1") + + assert result.succeeded + score_instances = { + s.label_dict["instance"] for s in result.samples if s.metric == "anomaly_score" + } + # host-b was scored even though host-a blew up. + assert "host-b" in score_instances + assert "host-a" not in score_instances + assert _collect( + ops, + "anomaly_failures_total", + labels={"detector": "", "reason": REASON_SERIES_EXCEPTION, "group": "g1"}, + ) == pytest.approx(1.0) + + +@pytest.mark.asyncio +async def test_baseline_band_emitted_when_detector_provides_it( + stub_source: StubSource, +) -> None: + df = clean_baseline(n=120) + stub_source.respond(lambda _: StubResponse(series=[make_series({"instance": "h"}, df)])) + + class BandDetector: + name: ClassVar[str] = "Band" + description: ClassVar[str] = "emits an explicit baseline band" + defaults: ClassVar[dict[str, Any]] = {} + + def fit_score(self, df: pd.DataFrame, *_: Any, **__: Any) -> pd.DataFrame: + ts = float(df["timestamp"].iloc[-1]) + return pd.DataFrame( + [ + { + "timestamp": ts, + "score": 0.0, + "baseline": 10.0, + "baseline_upper": 12.0, + "baseline_lower": 8.0, + "is_outside": False, + } + ] + ) + + registry.register(BandDetector) # type: ignore[arg-type] + try: + cfg = _config(detectors=[{"name": "Band"}]) + runner, _store, _ops = _runner(cfg, stub_source) + result = await runner.run_group("g1") + assert result.succeeded + by_metric = {s.metric: s for s in result.samples if s.metric.startswith("anomaly_baseline")} + assert "anomaly_baseline_upper" in by_metric + assert "anomaly_baseline_lower" in by_metric + assert float(by_metric["anomaly_baseline_upper"].value) == pytest.approx(12.0) + assert float(by_metric["anomaly_baseline_lower"].value) == pytest.approx(8.0) + finally: + registry.unregister("Band") + + +@pytest.mark.asyncio +async def test_baseline_band_omitted_when_detector_has_no_band( + stub_source: StubSource, +) -> None: + # MAD emits only a point baseline — the band metrics must not appear, + # so detectors that don't model a band never publish a bogus 0.0 edge. + df = clean_baseline(n=120) + stub_source.respond(lambda _: StubResponse(series=[make_series({"instance": "h"}, df)])) + cfg = _config(detectors=[{"name": "MAD"}]) + runner, _store, _ops = _runner(cfg, stub_source) + result = await runner.run_group("g1") + metrics = {s.metric for s in result.samples} + assert "anomaly_baseline" in metrics + assert "anomaly_baseline_upper" not in metrics + assert "anomaly_baseline_lower" not in metrics + + @pytest.mark.asyncio async def test_failing_query_does_not_stop_other_queries(stub_source: StubSource) -> None: good = clean_baseline(n=120) diff --git a/docs/detectors.md b/docs/detectors.md index f9077e2..2ef1751 100644 --- a/docs/detectors.md +++ b/docs/detectors.md @@ -740,6 +740,14 @@ values. ESD pass searches for. - `alpha` (default `0.05`) — significance level for the ESD test. +**Score scale.** The score is the ESD test statistic divided by its +critical value: ~1.0 means the latest residual sits exactly at the +`alpha` significance boundary, and higher is more extreme. This is **not** +the sigma-multiplier scale that `MAD`/`Hampel` use, so the universal +default `alert_thresholds.score: 3.0` is conservative here — it fires only +well past significance. To fire at the ESD significance level, set a lower +per-query (or per-detector) `alert_thresholds.score`, e.g. `1.0`. + **When to choose it.** Strongly seasonal signals where you want robust outlier detection that doesn't false-fire on the daily shape. More powerful than `HourOfDayMAD` for detecting subtle outliers in