From eaa793a0b80548b004bfe859892b444f7f804a93 Mon Sep 17 00:00:00 2001 From: Zilinghan Date: Tue, 7 Jul 2026 11:34:25 -0500 Subject: [PATCH 1/2] Update the documentation for drift detection to include general and config explanations --- docs/drift_detectors.md | 74 +++++++++++++++++++++++++++-------------- 1 file changed, 49 insertions(+), 25 deletions(-) diff --git a/docs/drift_detectors.md b/docs/drift_detectors.md index cbbce6c..205e7a3 100644 --- a/docs/drift_detectors.md +++ b/docs/drift_detectors.md @@ -60,11 +60,15 @@ Defined in `src/drift_detection/detectors/base.py`: ### `ADWINDetector` (`src/drift_detection/detectors/statistical_detectors.py`) +Brief Explanation: + +ADaptive WINdowing (ADWIN) keeps a variable-length sliding window of the monitored metric and cuts it in two whenever the means of the two sub-windows differ significantly; a cut signals drift and the window shrinks to the recent regime. **It handles both gradual and abrupt shifts, needs no fixed window size, and works well as the general-purpose default for a scalar such as loss or error rate.** + Constructor options: -- `delta` (config: `adwin_delta`) -- `minor_threshold` (config: `adwin_minor_threshold`) -- `moderate_threshold` (config: `adwin_moderate_threshold`) +- `adwin_delta`: confidence bound for a window cut. Lower = stricter test, so fewer/later detections and fewer false alarms; Higher = more sensitive. Typical range `0.001 - 0.01` (default `0.002`). +- `adwin_minor_threshold`: recent-drift-rate boundary below which drift maps to the `continual_learning` regime. Higher pushes more events into CL rather than fine-tuning. Range `0 - 1` (default `0.3`). +- `adwin_moderate_threshold`: drift-rate boundary above which drift escalates to `retrain`; between the two thresholds is `fine_tuning`. Must be `>= minor_threshold`; range `0 - 1` (default `0.6`). - `name` (constructor-only; not exposed in config) Behavior: @@ -75,13 +79,17 @@ Behavior: ### `KSWINDetector` +Brief Explanation: + +Kolmogorov-Smirnov WINdowing (KSWIN) runs a two-sample KS test comparing the most recent `kswin_stat_size` samples against a reference window of `kswin_window_size` samples, firing when the two distributions differ at significance level `kswin_alpha`. As it compares full distributions rather than just means, it catches shape/variance changes an averaging detector would miss, **making it a fit for cases where the distribution shifts without the mean moving much.** + Constructor options: -- `alpha` (config: `kswin_alpha`) -- `window_size` (config: `kswin_window_size`) -- `stat_size` (config: `kswin_stat_size`) -- `minor_threshold` (constructor default only; not in config loader call) -- `moderate_threshold` (constructor default only; not in config loader call) +- `kswin_alpha`: KS-test significance level. Lower = stricter, fewer false alarms but slower to react; Higher = more sensitive. Typical range `0.001 -0.01` (default `0.005`). +- `kswin_window_size`: total samples retained; the reference window is `window_size - stat_size`. Larger = more stable baseline but more memory and slower adaptation. Typically `50 - 300` (default `100`). +- `kswin_stat_size`: number of most-recent samples KS-tested against the reference. Larger = smoother but laggier detection. Must be `< window_size`; typically `20 - 50` (default `30`). +- `minor_threshold` (constructor default only; not in config loader call): drift-rate boundary for the `continual_learning` regime, `0 - 1` (default `0.3`). +- `moderate_threshold` (constructor default only; not in config loader call): drift-rate boundary for the `retrain` regime, `0 - 1` (default `0.6`). - `name` (constructor-only) Behavior: @@ -91,14 +99,18 @@ Behavior: ### `PageHinkleyDetector` +Brief Explanation: + +The Page-Hinkley test accumulates the running deviation of the metric from its historical mean and fires once that cumulative sum exceeds `ph_threshold`. It is cheap, low-memory, and quick to react, **making it a good fit for real-time monitoring of abrupt mean shifts.** + Constructor options: -- `min_instances` (config: `ph_min_instances`) -- `delta` (config: `ph_delta`) -- `threshold` (config: `ph_threshold`) -- `alpha` (config: `ph_alpha`) -- `minor_threshold` (constructor default only; not in config loader call) -- `moderate_threshold` (constructor default only; not in config loader call) +- `ph_min_instances`: warm-up samples observed before detection can fire. Higher = more stable baseline but slower first detection. Typically `20 - 100` (default `30`). +- `ph_delta`: slack subtracted from each deviation, i.e. the smallest change treated as meaningful. Higher = ignores small fluctuations (fewer false alarms); Lower = more sensitive. Typically `0.001 - 0.05` (default `0.005`). +- `ph_threshold`: cumulative-sum value that triggers drift. Higher = needs a larger/longer shift, so fewer/later detections; Lower = more sensitive. Scale depends on the metric; typically `10- 100` (default `50`). +- `ph_alpha`: forgetting factor for the running mean; closer to `1` weights history more (slower to adapt), lower forgets faster. Range just below `1`, e.g. `0.99 - 0.9999` (default `0.9999`). +- `minor_threshold` (constructor default only; not in config loader call): drift-rate boundary for the `continual_learning` regime, `0 - 1` (default `0.3`). +- `moderate_threshold` (constructor default only; not in config loader call): drift-rate boundary for the `retrain` regime, `0 - 1` (default `0.6`). - `name` (constructor-only) Behavior: @@ -108,14 +120,18 @@ Behavior: ### `ModelPerformanceDetector` (`src/drift_detection/detectors/model_performance_detector.py`) +Brief Explanation: + +A batch-level detector built on Evidently's `DataDriftPreset` that, rather than watching a single scalar, compares a reference dataset (data, predictions, targets captured when the model was healthy) against an incoming batch and flags drift when the share of drifted columns exceeds `drift_share_threshold`. This gives a richer, feature-aware signal but requires reference data up front, so use it for distribution-level analysis across many features (see the integration note below before wiring it into the monitor). + Constructor options: -- `reference_data` -- `reference_predictions` -- `reference_targets` -- `drift_share_threshold` -- `minor_threshold` -- `moderate_threshold` +- `reference_data`: baseline feature table (`pandas.DataFrame`) captured when the model was healthy; incoming batches are compared against it. +- `reference_predictions`: baseline model predictions aligned with `reference_data`; optional, enables prediction-drift analysis. +- `reference_targets`: baseline ground-truth labels; optional, enables target/performance-drift analysis. +- `drift_share_threshold`: fraction of columns that must be flagged as drifted before overall drift fires. Higher = more tolerant (fewer alarms), lower = more sensitive. Range `0 - 1` (Evidently default `0.5`). +- `minor_threshold`: drift-score boundary for the `continual_learning` regime, `0- 1` (default `0.3`). +- `moderate_threshold`: drift-score boundary for the `retrain` regime, `0- 1` (default `0.6`). - `name` Behavior modes: @@ -130,15 +146,19 @@ Integration note: ### `ModelEvalDetector` (`detector_name = "EvalDetector"`) +Brief Explanation: + +The most direct detector: on each update it re-evaluates the model via `modelHarness.eval()` and compares each metric against a reference value, respecting a per-metric `higher_is_better` flag, declaring drift when current performance falls short of the reference. It is a straightforward "has the model gotten worse?" check. **It is best to use when you have trustworthy reference validation metrics and want an interpretable signal.** + Constructor options: - `name` only. Expected `update(...)` kwargs: -- `modelHarness` -- `reference_validation_metrics` -- `higher_is_better` +- `modelHarness`: the harness whose `.eval()` is called to compute current validation metrics. +- `reference_validation_metrics`: baseline metric values to compare against; must be the same length/order as `modelHarness.eval()` output. +- `higher_is_better`: per-metric dict/flags stating direction of "good"; `True` means a drop below reference counts as drift, `False` means a rise above reference does. Integration note: @@ -147,10 +167,14 @@ Integration note: ### `EnsembleDetector` +Brief Explanation: + +Wraps several sub-detectors and combines their signals under a `voting` rule (`majority`, `unanimous`, `any`, or weighted) so you can trade sensitivity against false-alarm rate: e.g. `any` reacts to the first detector that fires while `unanimous` requires full agreement. It is conceptually useful for robustness, but note it is not currently loadable from config (see integration note). + Constructor options: -- `detectors: list[BaseDriftDetector]` -- `voting`: `majority`, `unanimous`, `any`, or weighted fallback +- `detectors: list[BaseDriftDetector]`: sub-detectors whose signals are combined; more detectors = more robust but more compute. +- `voting`: `majority`, `unanimous`, `any`, or weighted fallback: how signals combine. `any` = most sensitive (first firing wins), `majority` = balanced, `unanimous` = most conservative (all must agree). - `name` Integration note: From 35df38590a941edf864ce922e1db33800553f196 Mon Sep 17 00:00:00 2001 From: Zilinghan Date: Tue, 7 Jul 2026 14:42:02 -0500 Subject: [PATCH 2/2] add a new skill to help users to choose detector and hyperparameters --- .claude/skills/choose-detector/SKILL.md | 149 ++++++++++++++++++++++++ .codex/skills/choose-detector/SKILL.md | 127 ++++++++++++++++++++ 2 files changed, 276 insertions(+) create mode 100644 .claude/skills/choose-detector/SKILL.md create mode 100644 .codex/skills/choose-detector/SKILL.md diff --git a/.claude/skills/choose-detector/SKILL.md b/.claude/skills/choose-detector/SKILL.md new file mode 100644 index 0000000..2dafbae --- /dev/null +++ b/.claude/skills/choose-detector/SKILL.md @@ -0,0 +1,149 @@ +--- +name: choose-detector +description: | + Help the user pick a drift detector and tune its settings for an apeiron run. + Use when the user asks which detector to use, how to configure drift detection, + what ADWIN/KSWIN/PageHinkley/threshold values to set, or wants a ready-to-use + [drift_detection] config block. Asks a few questions about the monitored metric + and drift shape, recommends a detector, writes a filled-in TOML block, and + validates that it loads. Does NOT run a full experiment — for that use + explore-examples or custom-experiment. +argument-hint: "[config_path]" +user-invocable: true +allowed-tools: + - Bash + - Read + - Edit + - Write + - Grep + - Glob + - AskUserQuestion +--- + +Help the user choose a drift detector and produce a validated `[drift_detection]` +config block. The authoritative reference for detector behavior and every option +is `docs/drift_detectors.md` — read it first and keep this skill consistent with +it rather than restating numbers that may drift. + +## Arguments +- `$1`: Optional path to an existing config TOML. If given, patch its + `[drift_detection]` section in place. If omitted, emit a standalone block the + user can paste into their config. + +## Ground truth to respect (do not recommend around it) +Only three detectors are plug-and-play in the current `ContinuousMonitor` flow, +because `_check_drift()` calls `detector.update(agg_metric)` with a single +aggregated scalar: +- `ADWINDetector`, `KSWINDetector`, `PageHinkleyDetector`. + +The rest are **not** drop-in and must not be recommended as the default: +- `ModelPerformanceDetector` needs reference data + batch DataFrames (not passed by the monitor). +- `EvalDetector` (`ModelEvalDetector`) needs extra `update(...)` kwargs the monitor doesn't send. +- `EnsembleDetector` raises `NotImplementedError` in `load_drift_detector`. + +Confirm this is still true before relying on it: +```bash +sed -n '1,80p' src/apeiron/drift_detection/load_drift_detector.py +``` +If a user specifically wants one of the non-wired detectors, be honest that it +requires extra wiring and point them at the integration notes in the doc. + +## Procedure + +### 1. Understand the monitored signal +Ask the user (batch these with AskUserQuestion): +- **What metric** feeds the detector, and its rough scale? Bounded like accuracy + or error rate in `[0, 1]`, or unbounded like a loss? (This drives `metric_index` + and threshold scaling.) +- **What drift shape** do they expect? + - abrupt jumps in the mean + - gradual/slow drift + - distribution or variance change with little mean movement +- **Sensitivity vs. false alarms**: react early and tolerate some false alarms, + or only fire on clear, sustained drift? +- **Cadence**: roughly how many `update()` calls (i.e. `detection_interval`-sized + checks) happen before they'd want a first detection, and how many batches per + check. This sets warm-up expectations. + +Note: the scalar detectors fire on **change in either direction** — they do not +know "good" from "bad". If the user only cares about degradation, say so plainly +(that is what the non-wired `EvalDetector` is for). + +### 2. Recommend a detector +Map answers to a detector: +- distribution / variance / shape change without mean movement → **KSWIN** +- abrupt mean shift, want fast + cheap detection → **PageHinkley** +- gradual, mixed, or "not sure / general default" → **ADWIN** + +State the recommendation and the one-line reason. If it's a close call, name the +runner-up and the tradeoff. + +### 3. Recommend settings (scale to the metric) +Pull defaults and semantics from `docs/drift_detectors.md`. Adjust for the +metric scale and the sensitivity preference: +- **ADWIN** — `adwin_delta` is the main knob: lower (`~0.001`) = fewer false + alarms/later, higher (`~0.01`) = more sensitive. Keep `adwin_minor_threshold` + / `adwin_moderate_threshold` at `0.3` / `0.6` unless they want to steer the + CL→fine-tune→retrain regime split. +- **KSWIN** — `kswin_alpha` for sensitivity; size `kswin_window_size` / + `kswin_stat_size` to how many samples they can retain (`stat_size < window_size`). +- **PageHinkley** — `ph_threshold` scale **depends on the metric**: for a bounded + metric in `[0,1]` (accuracy/error) the default `50` is very large and will + rarely fire, so start much smaller (order `1–10`) and tune; for larger-magnitude + losses, larger thresholds are appropriate. `ph_delta` is the slack (min change + treated as real); `ph_min_instances` is warm-up. + +Set `detection_interval`, `aggregation` (`mean`/`median`/`last`), `metric_index`, +and `max_stream_updates` from the cadence answers. Explain any value that +deviates from the doc default. + +### 4. Produce the config block +Emit a complete, paste-ready section, e.g.: +```toml +[drift_detection] +detector_name = "ADWINDetector" +detection_interval = 10 +aggregation = "mean" +metric_index = 0 +reset_after_learning = false +max_stream_updates = 20 + +adwin_delta = 0.002 +adwin_minor_threshold = 0.3 +adwin_moderate_threshold = 0.6 +``` +If `$1` was given, patch that file's `[drift_detection]` section (Edit); keep the +keys that don't belong to the chosen detector untouched or drop the unused +detector-specific keys, matching the existing file style. + +### 5. Validate that it loads (no full run) +Build the config and instantiate the detector — this confirms the TOML parses, +the dataclass validates, and the detector name + params are accepted, without a +training run: +```bash +PYTHONPATH=src poetry run python -c " +from apeiron.config.configuration import build_config +from apeiron.drift_detection.load_drift_detector import load_drift_detector +cfg = build_config(['--config', '']) +d = load_drift_detector(cfg) +print('OK:', type(d).__name__) +print(cfg.drift_detection) +" +``` +(`PYTHONPATH=src` is required so `import apeiron` resolves — the package lives +under `src/apeiron` and `import apeiron` fails without it.) +For a standalone block (no `$1`), write it to a temp file first and validate that +(it still needs the other required sections `[model]`/`[data]`/`[train]` to build +a full `Config`; if the user has no config yet, validate against an example TOML +with `--set drift_detection.detector_name=...` overrides instead, or just confirm +the detector loads via `load_drift_detector` with an example config). + +Report the outcome plainly: recommended detector, why, the settings that differ +from defaults, and that the config loaded successfully. To actually observe +detection behavior, hand off to `explore-examples` or `custom-experiment`. + +## Notes +- Quick way to A/B a detector on a shipped example without editing files: + `poetry run python -m src.main --config examples/mnist/mnist.toml --set drift_detection.detector_name=PageHinkleyDetector --set drift_detection.ph_threshold=5` +- Keep `docs/drift_detectors.md` as the single source of truth; if you find this + skill and the doc disagree, fix the doc and follow it. diff --git a/.codex/skills/choose-detector/SKILL.md b/.codex/skills/choose-detector/SKILL.md new file mode 100644 index 0000000..dc871db --- /dev/null +++ b/.codex/skills/choose-detector/SKILL.md @@ -0,0 +1,127 @@ +--- +name: choose-detector +description: Help the user pick a drift detector and tune its settings for an apeiron run. Use when the user asks which detector to use, how to configure drift detection, what ADWIN/KSWIN/PageHinkley/threshold values to set, or wants a ready-to-use [drift_detection] config block. Asks a few questions about the monitored metric and drift shape, recommends a detector, writes a filled-in TOML block, and validates that it loads. Does not run a full experiment; for that use explore-examples or custom-experiment. +metadata: + short-description: Recommend and configure a drift detector +--- + +# Choose Detector + +Help the user choose a drift detector and produce a validated `[drift_detection]` +config block. The authoritative reference is `docs/drift_detectors.md`; read it +first and stay consistent with it rather than restating values that may change. + +## Inputs + +- Optional config path: if provided, patch that file's `[drift_detection]` + section in place. If omitted, emit a standalone block the user can paste in. + +## Ground Truth + +Only three detectors are drop-in for the current `ContinuousMonitor` flow, because +`_check_drift()` calls `detector.update(agg_metric)` with a single aggregated +scalar: + +- `ADWINDetector`, `KSWINDetector`, `PageHinkleyDetector`. + +The others are not drop-in and must not be offered as defaults: + +- `ModelPerformanceDetector` needs reference data and batch DataFrames. +- `EvalDetector` (`ModelEvalDetector`) needs extra `update(...)` kwargs the monitor does not send. +- `EnsembleDetector` raises `NotImplementedError` in the loader. + +Confirm before relying on this: + +```bash +sed -n '1,80p' src/apeiron/drift_detection/load_drift_detector.py +``` + +## Procedure + +### 1. Understand the Signal + +Ask the user: + +- Which metric feeds the detector, and its scale (bounded like accuracy/error in `[0, 1]`, or unbounded like a loss)? +- Expected drift shape: abrupt mean jumps, gradual drift, or distribution/variance change with little mean movement. +- Sensitivity preference: react early and tolerate false alarms, or fire only on clear sustained drift. +- Cadence: how many checks before a first detection is wanted, and how many batches per check. + +Note that the scalar detectors fire on change in either direction; they do not +distinguish good from bad. If the user only cares about degradation, say so +plainly (that is the non-wired `EvalDetector`'s job). + +### 2. Recommend a Detector + +- distribution/variance/shape change without mean movement: KSWIN +- abrupt mean shift, fast and cheap: PageHinkley +- gradual, mixed, or general default: ADWIN + +State the choice and a one-line reason; name the runner-up if it is close. + +### 3. Recommend Settings + +Pull defaults and semantics from `docs/drift_detectors.md` and scale to the metric: + +- ADWIN: `adwin_delta` is the main sensitivity knob; keep the two thresholds at defaults unless steering the regime split. +- KSWIN: `kswin_alpha` for sensitivity; size the windows to retained samples (`stat_size < window_size`). +- PageHinkley: `ph_threshold` scale depends on the metric. For a bounded metric in `[0,1]` the default `50` is very large and rarely fires, so start around `1-10` and tune; larger losses need larger thresholds. `ph_delta` is the slack, `ph_min_instances` is warm-up. + +Set `detection_interval`, `aggregation`, `metric_index`, and `max_stream_updates` +from the cadence answers, and explain any non-default value. + +### 4. Produce the Config Block + +Emit a complete, paste-ready section: + +```toml +[drift_detection] +detector_name = "ADWINDetector" +detection_interval = 10 +aggregation = "mean" +metric_index = 0 +reset_after_learning = false +max_stream_updates = 20 + +adwin_delta = 0.002 +adwin_minor_threshold = 0.3 +adwin_moderate_threshold = 0.6 +``` + +If a config path was given, patch its `[drift_detection]` section, matching the +existing file style. + +### 5. Validate (no full run) + +Build the config and instantiate the detector to confirm the TOML parses and the +detector accepts the params: + +```bash +PYTHONPATH=src poetry run python -c " +from apeiron.config.configuration import build_config +from apeiron.drift_detection.load_drift_detector import load_drift_detector +cfg = build_config(['--config', '']) +d = load_drift_detector(cfg) +print('OK:', type(d).__name__) +print(cfg.drift_detection) +" +``` +(`PYTHONPATH=src` is required so `import apeiron` resolves; the package lives +under `src/apeiron`.) + +For a standalone block, validate against an example TOML with `--set` overrides +(a full `Config` still needs `[model]`/`[data]`/`[train]`). Report the recommended +detector, the reason, the non-default settings, and that the config loaded. + +## Useful Commands + +A/B a detector on a shipped example without editing files: + +```bash +poetry run python -m src.main --config examples/mnist/mnist.toml \ + --set drift_detection.detector_name=PageHinkleyDetector \ + --set drift_detection.ph_threshold=5 +``` + +Keep `docs/drift_detectors.md` as the single source of truth; if the doc and this +skill disagree, fix the doc and follow it.