Some smaller performance/configuration like improvements

charts/promanomaly-metrics-adapter/README.md

@@ -1,96 +1,73 @@
-# promanomaly-metrics-adapter
+# promanomaly Metrics Adapter
 
-An **opt-in** Kubernetes external/custom metrics adapter for promanomaly.
+An **opt-in**, stateless Kubernetes external/custom metrics adapter for promanomaly.
 
-It is a stateless aggregated apiserver that re-serves the anomaly metrics
-the detector already wrote to the TSDB through
-`external.metrics.k8s.io` / `custom.metrics.k8s.io`, so your existing HPA
-and KEDA tooling can consume anomaly signal like any other metric.
+It re-exposes the anomaly metrics written by the detector to the TSDB through the standard Kubernetes `external.metrics.k8s.io` and `custom.metrics.k8s.io` APIs. This lets HPA and KEDA scale on anomaly signals exactly like any other metric.
 
-promanomaly stays the metrics **provider**; the autoscaler stays the
-**decision-maker**. The adapter never makes a reaction decision itself.
+promanomaly is the **metrics provider**. The autoscaler remains the **decision maker**.
 
-## Guard-rails (read first)
+## Guard-rails (Read First)
 
-Anomaly signal is a *sharp* scaling input. Wire it carefully:
+Anomaly signals are sharp scaling inputs. Use them carefully:
 
-- **Scale on `anomaly_density`**, not raw `anomaly_score`. "Add workers
-  when the backlog is anomalously deep" is sound; "scale on how
-  anomalous a latency metric looks" *amplifies* incidents — a latency
-  spike scales you up, which can deepen the spike.
-- **Gate on `anomaly_confidence_score` and `anomaly_duration_seconds`.**
-  React to sustained, high-confidence anomalies, not single-tick blips.
-- **Always pair an anomaly-driven scaler with a reactive fallback HPA**
-  (CPU/memory or queue length). If the detector or TSDB is unavailable,
-  the fallback keeps the workload safe.
+- **Scale on `anomaly_density`** (fraction of a group that is anomalous), not raw `anomaly_score`.
+- **Gate on `anomaly_severity`** (already includes confidence and duration).
+- **Always pair** an anomaly-driven scaler with a traditional reactive HPA fallback (CPU, memory, or queue depth).
 
-The default `metrics` list deliberately omits scaling presets for
-`anomaly_score`. Worked recipes with these guard-rails baked in live in
-[`examples/k8s/`](../../examples/k8s/).
+Safe configuration examples are in [`examples/k8s/`](../../examples/k8s/).
 
-## What it exposes
+## Exposed Metrics
 
-| Metric | Good for |
-| --- | --- |
-| `anomaly_density` | Fraction of a group currently anomalous — the recommended scaling input. |
-| `anomaly_severity` | Operator-facing 0-1 severity; gate scaling on it. |
-| `anomaly_active_series` | Count of firing series in a group. |
-| `anomaly_outside_threshold` | Per-series firing flag (0/1). |
+| Metric                        | Recommended Use                              |
+|-------------------------------|----------------------------------------------|
+| `anomaly_density`             | Primary scaling signal (fraction anomalous)  |
+| `anomaly_severity`            | 0–1 normalized severity for gating           |
+| `anomaly_active_series`       | Count of currently firing series             |
+| `anomaly_outside_threshold`   | Per-series 0/1 firing flag                   |
 
-Both API groups are served:
+Both API groups are supported:
+- `external.metrics.k8s.io` — recommended for KEDA and HPA `External` metrics.
+- `custom.metrics.k8s.io` — for HPA `Object` and `Pods` metrics.
 
-- **`external.metrics.k8s.io`** — cluster-scoped, the path KEDA's
-  `external` trigger and HPA `External` metric source use. Recommended.
-- **`custom.metrics.k8s.io`** — the same signals associated with objects
-  (pods, namespaces) for HPA `Object`/`Pods` rules.
-
-## Install
+## Quick Install
 
 ```bash
 helm install promanomaly-adapter charts/promanomaly-metrics-adapter \
   --namespace monitoring \
   --set datasource.url=http://victoriametrics.monitoring.svc:8428/
 ```
 
-Verify:
+Verify it is registered:
 
 ```bash
 kubectl get apiservices | grep metrics.k8s.io
+
 kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .
 ```
 
 ## TLS
 
-The aggregation layer only talks HTTPS to an APIService backend, so a
-serving cert is mandatory.
-
-- **Default**: the chart mints a self-signed CA + serving cert and wires
-  the APIService `caBundle` to it. Simple, but it **regenerates on every
-  `helm upgrade`** (a brief reconcile blip).
-- **Rotation-free**: set `tls.existingSecret` to a `kubernetes.io/tls`
-  Secret (e.g. from cert-manager) and `tls.caBundle` to its CA. No churn.
-- **Skip verification**: `apiService.insecureSkipTLSVerify: true` if you
-  accept the aggregation layer not verifying the adapter cert.
-
-## Key values
-
-| Key | Default | Description |
-| --- | --- | --- |
-| `datasource.url` | `http://victoriametrics:8428/` | TSDB the detector writes to. |
-| `datasource.auth.type` | `none` | `none`/`bearer`/`basic`/`mtls`. |
-| `datasource.auth.existingSecret` | `""` | Secret with the credential (keys: `token`/`password`/`ca.crt`+`tls.crt`+`tls.key`), mounted and referenced by file. |
-| `metrics` | the four above | Anomaly metrics to expose. |
-| `customResources` | `[pods, namespaces]` | Object kinds for custom metrics. |
-| `tls.existingSecret` | `""` | Use a managed serving cert instead of generating one. |
-| `apiService.enabled` | `true` | Register the two APIService objects. |
-| `apiService.insecureSkipTLSVerify` | `false` | Skip aggregation-layer cert verification. |
-| `rbac.create` | `true` | auth-delegator + auth-reader + HPA metrics-reader RBAC. |
-| `networkPolicy.enabled` | `true` | Restrict ingress to the serving port. |
-
-## How it differs from k8s-prometheus-adapter
-
-This adapter is purpose-built for the bounded set of anomaly metrics
-promanomaly emits, with the guard-rails documented above. It reads the
-same TSDB the detector writes to and translates label selectors directly
-into PromQL matchers. For arbitrary Prometheus metrics, use the
-general-purpose [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter).
+The Kubernetes aggregation layer requires HTTPS.
+
+The chart can:
+- Automatically generate a self-signed serving certificate (default, simple).
+- Use an existing `kubernetes.io/tls` Secret via `tls.existingSecret` (recommended for production, e.g. from cert-manager).
+
+## Key Configuration Values
+
+| Key                                 | Default                  | Description |
+|-------------------------------------|--------------------------|-------------|
+| `datasource.url`                    | —                        | URL of the TSDB written by the detector |
+| `datasource.auth.type`              | `none`                   | `none` / `bearer` / `basic` / `mtls` |
+| `datasource.auth.existingSecret`    | `""`                     | Secret containing credentials |
+| `metrics`                           | Core anomaly metrics     | Metrics to expose |
+| `customResources`                   | `[pods, namespaces]`     | Kubernetes objects for custom metrics |
+| `tls.existingSecret`                | `""`                     | Use a managed TLS Secret |
+| `apiService.insecureSkipTLSVerify`  | `false`                  | Skip TLS verification (not recommended in production) |
+| `networkPolicy.enabled`             | `true`                   | Restrict ingress |
+
+## How It Differs from prometheus-adapter
+
+This adapter is purpose-built and opinionated specifically for promanomaly’s anomaly metrics. It includes built-in guard-rails and directly translates label selectors into PromQL.
+
+For exposing arbitrary Prometheus metrics, use the general-purpose [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter) instead.

charts/promanomaly-stack/README.md

@@ -1,22 +1,29 @@
-# promanomaly-stack umbrella chart
+# promanomaly-stack Umbrella Chart
 
-Greenfield bundle: installs the [promanomaly](../promanomaly/README.md)
-detector plus a VictoriaMetrics single-node TSDB in one ``helm
-install``. Production users running an existing TSDB should install
-the ``promanomaly`` chart on its own and point its ``datasource.url``
-at their long-term TSDB instead.
+Greenfield bundle that installs the **promanomaly detector** + a **VictoriaMetrics single-node TSDB** in one `helm install`.
+
+> **Production note**: If you already have a VictoriaMetrics (or any Prometheus-compatible TSDB), install the plain [`promanomaly` chart](../promanomaly/README.md) instead and point `datasource.url` at your existing long-term store.
+
+## Quick Start
 
 ```bash
 helm repo add promanomaly https://esops-dev.github.io/promanomaly
+
 helm dependency update charts/promanomaly-stack
-helm install obs promanomaly/promanomaly-stack
+
+helm install obs promanomaly/promanomaly-stack \
+  --namespace monitoring \
+  --create-namespace
 ```
 
-Subchart values are nested under each subchart's name in
-``values.yaml``:
+## Configuration
+
+Subchart values are nested under each component:
 
 ```yaml
 promanomaly:
+  datasource:
+    url: http://{{ include "victoria-metrics-single.fullname" . }}:8428/
   groups:
     - name: example
       queries:
@@ -27,5 +34,10 @@ promanomaly:
 
 victoria-metrics-single:
   server:
-    retentionPeriod: 30
+    retentionPeriod: 30d          # adjust as needed
+    # ... other VM settings
 ```
+
+Full reference is in `charts/promanomaly-stack/values.yaml`.
+
+That’s it — one command gives you a working detector + storage for immediate testing or small-scale production. For larger setups, switch to the standalone `promanomaly` chart with your own TSDB.

charts/promanomaly/README.md

@@ -1,60 +1,45 @@
-# promanomaly Helm chart
+# promanomaly Helm Chart
 
-Installs the promanomaly detector — a single ``Deployment``, a
-``Service`` for ``/metrics``, a ``ConfigMap`` for the detector
-configuration, an optional ``ServiceMonitor`` for Prometheus Operator
-setups, an optional ``PrometheusRule`` carrying the reference alerts,
-and a ``NetworkPolicy`` that locks down ``/-/reload`` and the debug
-endpoints by default.
+This chart installs the promanomaly detector as a single `Deployment`, with a `Service` for `/metrics`, a `ConfigMap` for your configuration, an optional `ServiceMonitor`, an optional `PrometheusRule` with reference alerts, and a `NetworkPolicy` that locks down `/-/reload` and debug endpoints by default.
 
-Bring your own Prometheus/VictoriaMetrics. For a greenfield install
-that bundles VictoriaMetrics, see the ``promanomaly-stack`` umbrella.
+Bring your own Prometheus or VictoriaMetrics. For a full stack (including VictoriaMetrics), use the `promanomaly-stack` umbrella chart.
 
-## Quick start
+## Quick Start
 
 ```bash
 helm repo add promanomaly https://esops-dev.github.io/promanomaly
+
 helm install my-detector promanomaly/promanomaly \
-  --set datasource.url=http://my-victoriametrics:8428/ \
+  --set datasource.url=http://victoriametrics:8428/ \
   --values my-groups.yaml
 ```
 
-Where ``my-groups.yaml`` carries detector groups under ``groups:``.
-
-## Values
-
-See ``values.yaml`` and the schema in ``values.schema.json``. The
-notable bits:
-
-- ``replicaCount`` defaults to ``1``. Single-replica is the simplest
-  topology; HA mode (``highAvailability.enabled: true``) is supported
-  and adds Lease-based leader election + a Redis-backed snapshot
-  cache so multiple replicas serve the same ``/metrics`` snapshot.
-  See ``examples/configs/ha.yaml`` and ``examples/production/``.
-- ``existingConfigMap`` lets GitOps tooling (Argo CD, Flux) manage the
-  detector configuration out-of-band; the chart then skips its own
-  ConfigMap rendering.
-- ``datasource.auth`` supports ``none``/``bearer``/``basic``/``mtls``.
-  For anything other than ``none``, set
-  ``datasource.auth.existingSecret`` to a Secret carrying the credential
-  — the chart mounts it as files and references it by path so it never
-  lands in the ConfigMap. Expected Secret keys: ``token`` (bearer),
-  ``password`` (basic; set ``datasource.auth.username`` too), or
-  ``ca.crt``/``tls.crt``/``tls.key`` (mTLS).
-- ``networkPolicy.enabled`` defaults on; turn it off if your cluster
-  does not use NetworkPolicies.
-- ``serviceMonitor.honorLabels: true`` is the default and required
-  for anomaly labels to flow through to Prometheus unchanged.
-- ``prometheusRule.enabled`` is **off** by default — we don't ship
-  opinionated thresholds. The bundled rules mirror
-  ``examples/alerts/promanomaly-rules.yaml`` and expose
-  per-rule toggles + tunable thresholds so operators can adopt the
-  ones they want without templating from scratch.
-
-## Linting locally
+`my-groups.yaml` contains your detector groups (under the `groups:` key).
+
+## Key Configuration Options
+
+| Value                              | Default     | Description |
+|------------------------------------|-------------|-------------|
+| `replicaCount`                     | `1`         | Single replica is simplest and fully supported |
+| `highAvailability.enabled`         | `false`     | Enables leader election + Redis-backed snapshot cache for HA |
+| `existingConfigMap`                | —           | Use your own ConfigMap (GitOps-friendly) |
+| `datasource.auth`                  | `none`      | Supports `none` / `bearer` / `basic` / `mtls` |
+| `datasource.auth.existingSecret`   | —           | Secret for credentials (mounted as files) |
+| `networkPolicy.enabled`            | `true`      | Locks down sensitive endpoints |
+| `serviceMonitor.enabled`           | `false`     | Creates ServiceMonitor for Prometheus Operator |
+| `serviceMonitor.honorLabels`       | `true`      | Required for correct label propagation |
+| `prometheusRule.enabled`           | `false`     | Installs reference alerts (tunable thresholds) |
+
+Full reference in `values.yaml` + `values.schema.json`.
+
+## Local Linting
 
 ```bash
 helm dependency update charts/promanomaly
+
 helm lint --strict charts/promanomaly
+
 helm template charts/promanomaly | kubeconform -strict -summary -ignore-missing-schemas
 ```
+
+That’s it. The chart is deliberately minimal and production-ready out of the box. See the [production-readiness checklist](../production-checklist.md) for the full go-live steps.

detector/src/promanomaly/detectors/bocpd.py

@@ -256,13 +256,25 @@ def fit_score(
         # the cumulative mass at the recent run-lengths — the standard
         # way to surface the change-point signal from a BOCPD posterior.
         recent_lag = min(lag + 1, weights.size)
-        # Clamp into [0, 1] — the weights *should* be a normalised
-        # probability distribution, but the underlying floating-point
-        # logsumexp + scipy.special.gammaln path occasionally returns
-        # 1.0 + 1e-16 due to round-off. Clamping keeps the published
-        # score within the documented contract without changing the
-        # firing decision (the threshold is far from the edges).
-        cp_prob = float(min(1.0, max(0.0, weights[:recent_lag].sum())))
+        # Insufficient-points guard. The run-length posterior holds one
+        # more state than the number of observed samples, so when the
+        # window has no more than ``lag`` samples (``weights.size <=
+        # lag + 1``) the "recent" window already spans every run length.
+        # The cumulative mass is then trivially ~1.0 and would fire a
+        # spurious change-point on any signal — there is no older tail to
+        # contrast the recent mass against. Report no change-point in that
+        # regime, mirroring the other detectors' zero-score behaviour on
+        # too-short input. (Normally masked by ``min_points`` >> ``lag``.)
+        if weights.size <= lag + 1:
+            cp_prob = 0.0
+        else:
+            # Clamp into [0, 1] — the weights *should* be a normalised
+            # probability distribution, but the underlying floating-point
+            # logsumexp + scipy.special.gammaln path occasionally returns
+            # 1.0 + 1e-16 due to round-off. Clamping keeps the published
+            # score within the documented contract without changing the
+            # firing decision (the threshold is far from the edges).
+            cp_prob = float(min(1.0, max(0.0, weights[:recent_lag].sum())))
         # MAP-style baseline: weighted average of regime means by their
         # posterior run-length mass. Falls back to the overall mean if
         # the posterior is degenerate (all zero mass).

detector/src/promanomaly/detectors/distribution_shift.py

@@ -114,8 +114,11 @@ def fit_score(
             mad = float(np.median(np.abs(baseline - baseline_median)))
             score = dist / mad if mad > 0.0 else 0.0
         elif statistic == "energy":
-            # Energy distance approximation via sorted samples.
-            dist = float(stats.wasserstein_distance(baseline, recent))
+            # Energy distance: sensitive to differences across the whole
+            # distribution (not just the optimal-transport cost Wasserstein
+            # measures), normalised by the baseline MAD like the
+            # Wasserstein branch so the two share a score scale.
+            dist = float(stats.energy_distance(baseline, recent))
             mad = float(np.median(np.abs(baseline - baseline_median)))
             score = dist / mad if mad > 0.0 else 0.0
         else:

detector/src/promanomaly/detectors/seasonal_hybrid_esd.py

@@ -8,10 +8,16 @@
 This is still baseline detection, not forecasting: it scores residuals
 against historical structure, it does not predict future values.
 
-Score is the ratio of the ESD test statistic to the critical value —
-values above 1.0 indicate the residual is an outlier at the configured
-significance level. Normalised so the default
-``alert_thresholds.score=3.0`` fires on clearly significant outliers.
+Score is the ratio of the ESD test statistic to the critical value: a
+score of ~1.0 means the latest residual sits exactly at the configured
+``alpha`` significance boundary, and higher scores are progressively more
+extreme outliers. Note this scale differs from the sigma-multiplier scale
+of MAD/Hampel — the universal default ``alert_thresholds.score=3.0`` is
+deliberately conservative here (it fires only well past the significance
+boundary), so groups that want firing at the ESD significance level should
+set a lower per-query/per-detector ``alert_thresholds.score`` (e.g. ~1.0).
+The detector leaves ``is_outside`` to the runner's threshold check rather
+than asserting its own significance verdict.
 """
 
 from __future__ import annotations