esops-dev · dj-wasabi · May 29, 2026 · May 29, 2026
@@ -22,7 +22,7 @@ Everything that shapes the data flowing into the forecasting model.
 - [Change-points](inputs/change-points.md) — input-side break detection and lookback adaptation
 - [Seasonality](inputs/seasonality.md) — periodicity detection
 - [Cold-start](inputs/cold-start.md) — sparse-history behaviour and peer-sibling borrowing
-- [Regressors](inputs/regressors.md) — calendar, custom PromQL, and holiday regressors, plus conditional forecasts (baseline given the current regressor state)
+- [Regressors](inputs/regressors.md) — calendar, custom PromQL, holiday, webhook (external pull), and calendar-file regressors, plus conditional forecasts (baseline given the current regressor state) and HMAC-signed webhooks
 - [Series discovery](inputs/discovery.md) — template-driven auto-fan-out over label values
 
 ## Outputs
@@ -43,6 +43,12 @@ metrics, calibration, explainability, and ad-hoc what-if endpoints.
 - [Canary forecasts](outputs/canary.md)
 - [Predictive autoscaling](predictive-autoscaling.md) — Kubernetes custom-metrics adapter that exposes forecast metrics to HorizontalPodAutoscaler for scaling on the projected load curve
 
+## Integrations
+
+- [Canary deploy forecast comparison](canary-analysis.md) — fit the baseline pods, project what the canary should look like, and emit `<id>_canary_forecast_deviation` for Argo Rollouts / Flagger AnalysisRun gates
+- [Forecast-driven webhook triggers](triggers.md) — push structured forecast events (predicted value, band, model, drift) to a webhook when a PromQL condition transitions true; optional HMAC signing
+- [Grafana annotations](integrations/grafana-annotations.md) — post significant forecast events (drift, change-point, deploy-risk, predicted breach, deviation) as Grafana annotations on the dashboards you already watch
+
 ## Operations
 
 Running the forecaster in production — diagnosis, capacity safeguards,

@@ -0,0 +1,76 @@
+# Canary deploy forecast comparison
+
+When you run a canary deploy (a small slice of traffic to a new version), you want to know whether the canary is behaving like the stable fleet *would have* under the same conditions.  
+
+promforecast can model that: it fits the **baseline** pods’ history, projects what the canary *should* look like, and surfaces the divergence between the canary’s actuals and that baseline-predicted forecast as a clear, alertable signal.
+
+This turns canary analysis into a data-driven comparison against a quality forecast, rather than manual eyeballing or hand-rolled thresholds.
+
+> **Note:** this is distinct from [canary / weighted sampling](outputs/canary.md), which fits only a fraction of a group’s series to save CPU.
+
+## Configuring
+
+Add a `canary:` block to a query. The selectors partition the already-fetched series into canary and baseline populations:
+
+```yaml
+groups:
+  - name: checkout
+    queries:
+      - id: http_requests
+        promql: sum by (version, pod) (rate(http_requests_total[5m]))
+        canary:
+          enabled: true
+          canary_selector: '{version="canary"}'
+          baseline_selector: '{version!="canary"}'
+```
+
+- `canary_selector` / `baseline_selector` are Prometheus-style label matchers (`=` and `!=` only). Regex (`=~`) is rejected at load time.
+- A series matching neither selector is ignored; matching both is a config error.
+
+## Emitted metrics
+
+For each canary series, the forecaster emits (in the same shape as existing deviation metrics):
+
+```
+<id>_canary_forecast_deviation{deployment, replica, version, model, level, ...}
+<id>_canary_forecast_deviation_outside_band{...}
+```
+
+- `<id>_canary_forecast_deviation` — signed normalised distance of the canary actual from the centre of the baseline-predicted band: `(actual - yhat) / (yhat_upper - yhat_lower)`.
+- `<id>_canary_forecast_deviation_outside_band` — `1` when the canary actual is outside the baseline band, else `0`.
+
+The labels preserve the canary series’ own labels plus the `model` that produced the baseline forecast and the confidence `level`.
+
+## Use with rollout controllers
+
+The emission shape is directly compatible with Argo Rollouts and Flagger.
+
+**Argo Rollouts** (`AnalysisTemplate`):
+
+```yaml
+metrics:
+  - name: canary-divergence
+    successCondition: result[0] <= 0.5
+    provider:
+      prometheus:
+        address: http://prometheus.monitoring:9090
+        query: max(http_requests_canary_forecast_deviation{version="canary"})
+```
+
+**Flagger** (`MetricTemplate`):
+
+```yaml
+spec:
+  provider:
+    type: prometheus
+    address: http://prometheus.monitoring:9090
+  query: max(http_requests_canary_forecast_deviation_outside_band{version="canary"})
+```
+
+Gate the rollout on the deviation staying inside the band (or the ratio staying below an operator-chosen magnitude).
+
+## Trade-off: baseline history required
+
+Canary analysis depends on the baseline pods having stable history to fit on. For a brand-new service with no baseline, fall back to [cold-start sibling borrowing](inputs/cold-start.md) until the stable fleet has enough history.
+
+If either population is empty or the baseline is too short to fit, no comparison is emitted that tick — the normal forecast path is unaffected.
@@ -78,6 +78,98 @@ If the lookback window contains no holidays, falls back to a single binary colum
 
 Supported countries/regions are whatever the installed `holidays>=0.50` package provides (≥150 countries + most major subdivisions).
 
+## Webhook regressors
+
+For a signal that does not live in Prometheus and is not on the built-in calendar list — active incidents from PagerDuty, deploys-in-flight from a CI/CD API, a custom internal "promotion-day" calendar — the `webhook` type lets the forecaster pull it in at fit time without standing up a Prometheus exporter or modifying the forecaster.
+
+```yaml
+regressors:
+  - id: active_incidents
+    type: webhook
+    url: https://pagerduty-bridge.internal/incident-count
+    timeout: 5s                  # default
+    cache_ttl: 1h                # default — reuse responses per (url, query, range)
+    headers:                     # optional static auth
+      Authorization: Bearer ${PAGERDUTY_TOKEN}
+    required: false              # default
+```
+
+The forecaster POSTs to `url` once per primary query (not per series) with a JSON body:
+
+```json
+{"query_id": "<id>", "regressor_id": "active_incidents", "start": "<ISO-8601>", "end": "<ISO-8601>"}
+```
+
+and expects a JSON time series back:
+
+```json
+{"timestamps": [1735689600, 1735693200], "values": [2, 1]}
+```
+
+Timestamps may be epoch seconds or ISO-8601 strings. The result is aligned to the primary series with the same nearest-asof join as a custom PromQL regressor, and cached per `(url, query_id, lookback)` for `cache_ttl` so a query returning many series — or a short refresh interval — doesn't hammer the external service.
+
+The forecaster stays stateless: the webhook owns whatever state it queries. Webhook failures (timeout, non-2xx, unparseable body) flow through the same graceful-degradation path as any other regressor — the forecast is fit *without* the regressor and `forecast_regressor_failures_total{…, reason="webhook_failure"}` increments. Set `required: true` to skip the series instead.
+
+`headers` values support `${ENV_VAR}` expansion against the process environment (injected from a Kubernetes Secret via `secretKeyRef`), so `Authorization: Bearer ${PAGERDUTY_TOKEN}` is resolved at request time rather than sent literally — keep the token out of the config. An unset variable expands to an empty string.
+
+> The shared webhook HTTP client is created when *any* query configures a `webhook` regressor. A config reload that adds the first webhook regressor wires the client on the fly, so no restart is required.
+
+### Example integrations
+
+**PagerDuty incident lookup** — a small bridge service queries the PagerDuty API for the count of active incidents in the requested window and returns it as a step series. Spikes in incidents become an exogenous input the model can attribute load changes to.
+
+**CI/CD deploy lookup** — a bridge to your CI/CD API (GitHub Actions, Argo, Jenkins) returns 1 on timestamps where a deploy completed. This is the dynamic sibling of the static `calendar_file` regressor below: use the webhook when "did a deploy happen?" is only answerable at query time, the calendar file when it's a known fixed schedule.
+
+### HMAC-verified responses
+
+When the webhook bridge signs its response body, add a `signing` block and the forecaster verifies it before accepting the data:
+
+```yaml
+regressors:
+  - id: active_incidents
+    type: webhook
+    url: https://pagerduty-bridge.internal/incident-count
+    signing:
+      algorithm: hmac-sha256     # only value today
+      secret_ref: ${WEBHOOK_SECRET}
+```
+
+The forecaster recomputes `HMAC-SHA256(secret, response_body)` and compares it constant-time against the `X-Promforecast-Signature: sha256=<hex>` header the bridge sets. A missing or mismatched signature is treated as a regressor failure with `reason="signature_invalid"` (same graceful-degradation path). `secret_ref` resolves a `${ENV_VAR}` reference against the process environment (injected from a Kubernetes Secret via `secretKeyRef`) or accepts a literal for local testing; the same `signing` block is available on [webhook triggers](../triggers.md) for the outbound direction.
+
+## Calendar-file regressors
+
+For a static internal calendar — release freezes, fiscal-quarter ends, marketing-campaign launches — the `calendar_file` type reads a YAML or iCal file mounted into the pod, without needing a webhook just to read a file.
+
+```yaml
+regressors:
+  - id: release_freeze
+    type: calendar_file
+    path: /etc/promforecast/calendars/release-freezes.yaml
+    expand: false                # default — single binary column
+```
+
+The file format is YAML (preferred, git-friendly) with an `entries:` list:
+
+```yaml
+# calendars/release-freezes.yaml
+entries:
+  - start: 2026-01-01
+    end: 2026-01-05
+    name: new-year-freeze
+  - start: 2026-12-20
+    end: 2026-12-31
+    name: holiday-freeze
+```
+
+or iCal (`.ics`, for tooling compatibility) — each `VEVENT` becomes one window from its `DTSTART` / `DTEND` / `SUMMARY`.
+
+The emission shape matches the `holidays` regressor:
+
+- `expand: false` (default) emits a single binary column `release_freeze` (1.0 on any day inside any window, 0.0 otherwise).
+- `expand: true` emits one column per distinct entry `name` (`release_freeze__new_year_freeze`, …).
+
+The calendar file is re-read automatically when its mtime changes, so a ConfigMap update to the mounted file is picked up on the next fit — the same effect as the main config's configmap-watch reload path, without a process restart. A missing or malformed file degrades gracefully with `reason="calendar_file_error"`.
+
 ## Failure handling
 
 | Failure type                        | Reason label      | `required: true` behaviour          |
@@ -88,6 +180,9 @@ Supported countries/regions are whatever the installed `holidays>=0.50` package
 | No datasource (dry-run / test)      | `no_source`       | Entire series skipped               |
 | `holidays` package missing          | `invalid_type`    | Series skipped                      |
 | Unknown country / region            | `invalid_type`    | Series skipped                      |
+| Webhook timeout / error / bad body  | `webhook_failure` | Entire series skipped               |
+| Webhook HMAC verification failed    | `signature_invalid` | Entire series skipped             |
+| Calendar file missing / malformed   | `calendar_file_error` | Series skipped                  |
 
 Optional regressors (default) only increment the failure counter; the forecast still emits without that feature.
 

@@ -0,0 +1,56 @@
+# Grafana annotations for forecast events
+
+If you live in Grafana, you want significant forecast events to appear *on the graph* — a vertical marker when a change-point is detected, drift threshold crossed, or a threshold breach is predicted. This lets you correlate forecasts with operational events without manually overlaying timelines.
+
+When enabled, the forecaster POSTs to Grafana’s annotations API on every enabled event that newly fires. Works with both Grafana OSS and Grafana Cloud.
+
+## Configuring
+
+Top-level `annotations:` block:
+
+```yaml
+annotations:
+  enabled: true
+  grafana_url: https://grafana.example.com
+  api_token_ref: ${GRAFANA_API_TOKEN}     # resolved from environment
+  events:
+    - drift
+    - change_point
+    - deploy_risk
+    - threshold_breach_predicted
+    - deviation_outside_band
+  timeout: 5s                              # default
+```
+
+- `grafana_url` is the base URL; posts go to `<grafana_url>/api/annotations`.
+- `api_token_ref` resolves a `${ENV_VAR}` (Grafana service-account token, typically from a Kubernetes Secret).
+- `events` selects which families post annotations. Empty list = none.
+
+## Events
+
+| `event_type`                  | Fires when… |
+|-------------------------------|-------------|
+| `drift`                       | Drift score crosses alert threshold |
+| `change_point`                | Preprocessing detects a change-point |
+| `deploy_risk`                 | Deploy-anchored risk score is active |
+| `threshold_breach_predicted`  | A `*_forecast_will_breach` sample predicts a breach |
+| `deviation_outside_band`      | Actual lands outside its forecast band |
+
+Each annotation carries a millisecond timestamp, human-readable text, and tags including `promforecast`, the `event_type`, and `group:`/`id:` scoping tags.
+
+## Rising-edge deduplication
+
+Sustained conditions post **once** on the rising edge, not every refresh. When the condition clears and re-fires, a new annotation is posted. This keeps dashboards readable.
+
+## Delivery telemetry
+
+```
+forecast_annotation_posts_total{event_type, outcome}   # outcome = success | http_error | timeout
+```
+
+Delivery failures never block fits — they are logged and the run continues.
+
+## Operational notes
+
+- Annotations are built at process start. Changes to `enabled`, `events`, or `grafana_url` require a restart.
+- Rising-edge state is in-process. After restart (or HA leader failover), still-active events re-post once — acceptable for annotations that mark moments rather than continuous state.
@@ -48,7 +48,7 @@ No `helm upgrade` needed; the rolling reload picks up the change via the ConfigM
 
 ## Stability commitments (from `v1` onward)
 
-- Top-level keys (`datasource`, `server`, `safety`, `defaults`, `sink`, `highAvailability`, `query_cache`, `telemetry`, `groups`) are part of the stable contract — no removals or renames.
+- Top-level keys (`datasource`, `server`, `safety`, `defaults`, `sink`, `highAvailability`, `query_cache`, `telemetry`, `groups`, `triggers`, `annotations`) are part of the stable contract — no removals or renames.
 - Field defaults may change in minor releases **only** when the new default is safer.
 - Behavioural defaults that operators rely on (e.g. `defaults.accuracy.evaluate: true`) are pinned.
 - New optional fields are additive and never bump the `apiVersion`.

@@ -0,0 +1,89 @@
+# Forecast-driven webhook triggers
+
+Triggers let the forecaster **push** a structured event to a webhook when a PromQL condition over forecast metrics becomes true — e.g. auto-archive cleanup when a disk-fill ETA drops below a day, a Slack post when error-budget burn accelerates, or a CI job when drift is detected.
+
+**How this differs from alternatives**:
+- vs. **Alertmanager / PrometheusRule** — the payload carries full forecast context (matched series, values, model identity, band/drift labels), not just “an alert fired”.
+- vs. **[webhook regressors](inputs/regressors.md#webhook-regressors)** — the direction is reversed: regressors *pull* signals in; triggers *push* events out.
+
+## Configuring
+
+Top-level `triggers:` list:
+
+```yaml
+triggers:
+  - name: disk-fill-imminent
+    condition: 'node_filesystem_avail_bytes_forecast{horizon="24h"} < 0'
+    webhook_url: https://hooks.slack.com/services/XXX/YYY/ZZZ
+    repeat_interval: 1h          # default
+    timeout: 5s                  # default
+    headers:                     # optional static auth
+      Authorization: Bearer ${SLACK_TOKEN}
+    payload_template: |
+      {"text": "Disk fill predicted within 24h (value $value)"}
+```
+
+- `condition` is a PromQL expression evaluated after group runs (re-evaluations that land within a few seconds of each other — e.g. several groups finishing one refresh cycle together — are coalesced into a single instant query, so trigger evaluation doesn't scale with group count).
+- The trigger fires on false → true transitions. Sustained-true conditions re-fire at most once per `repeat_interval`.
+
+## Payload
+
+With no `payload_template`, the forecaster POSTs:
+
+```json
+{
+  "name": "disk-fill-imminent",
+  "condition": "...",
+  "fired_at": "2026-05-29T12:00:00+00:00",
+  "value": -1.2e9,
+  "labels": {"instance": "node-7", "horizon": "24h", "model": "AutoARIMA"},
+  "samples": [{"labels": {...}, "value": -1.2e9}]
+}
+```
+
+`value` / `labels` come from the first matched series; `samples` carries the full matched vector.
+
+### Templating
+
+`payload_template` supports `$`-style placeholders:
+
+```yaml
+payload_template: '{"text": "$name fired: value=$value"}'
+```
+
+Available variables: `$name`, `$condition`, `$value`, `$fired_at`, `$samples_json`.
+
+## Authentication
+
+Static headers use the `headers:` block. Header values support `${ENV_VAR}` expansion against the process environment (injected from a Kubernetes Secret via `secretKeyRef`), so `Authorization: Bearer ${SLACK_TOKEN}` is resolved at delivery time rather than sent literally. An unset variable expands to an empty string.
+
+For end-to-end verification, add a signing block:
+
+```yaml
+signing:
+  algorithm: hmac-sha256
+  secret_ref: ${TRIGGER_SIGNING_SECRET}
+```
+
+The forecaster adds `X-Promforecast-Signature: sha256=<hex>`. Receivers recompute the HMAC over the raw body.
+
+## Delivery telemetry
+
+```
+forecast_trigger_fires_total{name, outcome}   # outcome = success | http_error | timeout
+```
+
+Delivery failures never block fits or other triggers.
+
+## Example receivers
+
+**Slack** — point `webhook_url` at a Slack incoming-webhook and use a simple `payload_template`.
+
+**Generic JSON receiver** — leave `payload_template` empty and verify the signature if configured.
+
+## Operational notes
+
+- Triggers are built at process start. Config changes require a restart.
+- Conditions are evaluated after each group run.
+- One-cycle lag is possible (TSDB ingestion).
+- Evaluation or delivery errors are logged and never block the fit pipeline.
@@ -41,6 +41,25 @@ groups:
             type: calendar
           - id: is_weekend
             type: calendar
+          # Optional webhook regressor: pull "deploys in flight" from a
+          # CI/CD bridge that isn't in Prometheus. Optional by default, so
+          # if the bridge is down the forecast still runs (the failure is
+          # counted in forecast_regressor_failures_total). The signing
+          # block is optional — drop it if the bridge doesn't sign.
+          - id: deploys_in_flight
+            type: webhook
+            url: https://cicd-bridge.internal/deploys-in-flight
+            cache_ttl: 15m
+            headers:
+              Authorization: Bearer ${CICD_BRIDGE_TOKEN}
+            signing:
+              algorithm: hmac-sha256
+              secret_ref: ${WEBHOOK_SIGNING_SECRET}
+          # Static internal calendar mounted into the pod: 1 during a
+          # release freeze, 0 otherwise. See examples/configs/calendars/.
+          - id: release_freeze
+            type: calendar_file
+            path: /etc/promforecast/calendars/release-freezes.yaml
 
       - id: http_5xx_per_second
         promql: |