Documentation and Skills for Drift Detectors

[Zilinghan]

Summary

In this PR, I have add explanations for each drift detectors, and each parameter for the detector and the recommended value ranges. I also add a new skill choose-detector to guide users on choosing the right detector and detector settings for their usecase.

Motivation & Context

Currently, it's not clear how to choose detector and how to choose a value for the detector settings, therefore, I added more information to drift_detectors.md and created a new skill.

Approach

Screenshots / Logs (optional)

Logs for /choose-detector skill

❯ /choose-detector

⏺ User answered Claude's questions:
  ⎿  · What metric will feed the drift detector, and roughly what scale is it on? → Accuracy
     / error in [0,1]
     · What kind of drift do you expect in that metric? → Distribution / variance
     · How should the detector trade off reacting early vs. avoiding false alarms? → Only
     clear drift
     · How often does the monitor check for drift, and how soon do you want a first
     detection? → Infrequent / stable

⏺ Based on your answers (bounded accuracy/error, distribution/variance change,
  only-clear-drift, infrequent/stable checks):

  Detector: KSWINDetector — a distribution/variance shift with little mean movement is
  precisely what a Kolmogorov–Smirnov windowing test catches and a mean-based detector
  (ADWIN/PageHinkley) would miss. Bonus: since your metric is bounded in [0,1], KSWIN has no
  threshold-scaling concerns.

  One honest caveat about "distribution/variance" here. The monitor feeds KSWIN a single
  aggregated scalar per check (aggregation of the metric buffer), so KSWIN tests the
  distribution of that per-check metric over time — not the per-batch feature distribution.
  That's still the right tool among the wired detectors, but two consequences:
  - If you aggregate by mean, within-check variance is averaged away before KSWIN ever sees
  it. To preserve the distributional/variance signal I recommend aggregation = "last" (raw
  per-check value) rather than mean.
  - If you truly want per-feature batch-distribution drift, that's ModelPerformanceDetector
  — which isn't wired into the monitor (needs reference data + batch DataFrames; see the
  doc's integration note).

  Settings — tuned for "only clear drift" + "infrequent/stable":
  - kswin_alpha = 0.001 (stricter than default 0.005) — fewer false alarms, only clear
  drift.
  - aggregation = "last" (changed from mean) — keeps the distributional signal instead of
  smoothing it.
  - detection_interval = 20 (raised from 10) — infrequent checks / stable baseline.
  - kswin_window_size = 100, kswin_stat_size = 30 — defaults (solid baseline without
  over-lengthening warm-up).

  ⚠️  Two things to confirm: metric_index = 0 must point at your accuracy/error metric in the
  harness's eval_metrics order; and KSWIN needs ~`window_sizechecks to warm up, so on a
  short run (fewupdate()calls givendetection_interval=20×max_stream_updates=20) consider
  lowering kswin_window_size` so the reference window actually fills.

  [drift_detection]
  detector_name = "KSWINDetector"
  detection_interval = 20
  aggregation = "last"
  metric_index = 0
  reset_after_learning = false
  max_stream_updates = 20

  kswin_alpha = 0.001
  kswin_window_size = 100
  kswin_stat_size = 30

  Now validating it loads (no config path given, so against a shipped example with --set
  overrides matching the block):

⏺ Bash(PYTHONPATH=src poetry run python -c "
      from apeiron.config.configuration import build_config…)
  ⎿  OK loaded: KSWINDetector
     interval=20 agg=last metric_index=0 alpha=0.001 window=100 stat=30

⏺ Validated ✓ — config parses, DriftDetectionCfg builds, and KSWINDetector instantiates with
  exactly these params (interval=20 agg=last metric_index=0 alpha=0.001 window=100
  stat=30).

  Summary

  - Detector: KSWINDetector — the only wired detector that catches distribution/variance
  shifts without mean movement; no threshold scaling needed for your bounded metric.
  - Non-default settings: kswin_alpha=0.001 (stricter, only-clear-drift), aggregation="last"
  (preserve distributional signal), detection_interval=20 (infrequent/stable). Window sizes
  left at defaults.
  - Config loads cleanly.

  Before you rely on it

  - Confirm metric_index=0 points at your accuracy/error metric.
  - Watch warm-up: KSWIN needs ~`window_size(100) checks to fill its reference window;
  withdetection_interval=20andmax_stream_updates=20a short run may not get there —
  dropkswin_window_size` (e.g. to 50) if you see it never firing.
  reference data + batch DataFrames; see the doc's integration note).

  Settings — tuned for "only clear drift" + "infrequent/stable":
  - kswin_alpha = 0.001 (stricter than default 0.005) — fewer false alarms, only clear drift.
  - aggregation = "last" (changed from mean) — keeps the distributional signal instead of smoothing it.
  - detection_interval = 20 (raised from 10) — infrequent checks / stable baseline.
  - kswin_window_size = 100, kswin_stat_size = 30 — defaults (solid baseline without over-lengthening warm-up).

  ⚠️  Two things to confirm: metric_index = 0 must point at your accuracy/error metric in the harness's eval_metrics order; and KSWIN
  needs ~`window_sizechecks to warm up, so on a short run (fewupdate()calls givendetection_interval=20×max_stream_updates=20) consider
  lowering kswin_window_size` so the reference window actually fills.

  [drift_detection]
  detector_name = "KSWINDetector"
  detection_interval = 20
  aggregation = "last"
  metric_index = 0
  reset_after_learning = false
  max_stream_updates = 20

  kswin_alpha = 0.001
  kswin_window_size = 100
  kswin_stat_size = 30

  Now validating it loads (no config path given, so against a shipped example with --set overrides matching the block):

⏺ Bash(PYTHONPATH=src poetry run python -c "
      from apeiron.config.configuration import build_config…)
  ⎿  OK loaded: KSWINDetector
     interval=20 agg=last metric_index=0 alpha=0.001 window=100 stat=30

⏺ Validated ✓ — config parses, DriftDetectionCfg builds, and KSWINDetector instantiates with exactly these params (interval=20 agg=last
  metric_index=0 alpha=0.001 window=100 stat=30).

  Summary

  - Detector: KSWINDetector — the only wired detector that catches distribution/variance shifts without mean movement; no threshold
  scaling needed for your bounded metric.
  - Non-default settings: kswin_alpha=0.001 (stricter, only-clear-drift), aggregation="last" (preserve distributional signal),
  detection_interval=20 (infrequent/stable). Window sizes left at defaults.
  - Config loads cleanly.

  Before you rely on it

  - Confirm metric_index=0 points at your accuracy/error metric.
  - Watch warm-up: KSWIN needs ~`window_size(100) checks to fill its reference window;
  withdetection_interval=20andmax_stream_updates=20a short run may not get there — dropkswin_window_size` (e.g. to 50) if you see it
  never firing.
  - Remember KSWIN fires on change in either direction (not just degradation).

  Next

  Hand off to explore-examples or custom-experiment to watch it detect on real data. Quick A/B without editing files:
  poetry run python -m src.main --config examples/mnist/mnist.toml \
    --set drift_detection.detector_name=KSWINDetector \
    --set drift_detection.aggregation=last \
    --set drift_detection.kswin_alpha=0.001

API / CLI Changes

NA

Breaking Changes

• None

Performance (optional)

Case	Before	After	Notes
foo()	123 ms	88 ms	Median of 50 runs

Security & Privacy

• No secrets committed
• Input validation added where needed

Dependencies

None

Testing Plan

• Unit tests
• Integration tests
• e2e / smoke test
• Manual steps: python -m app --help

Documentation

• Docstrings updated
• User docs / README updated
• CHANGELOG entry

Checklist

• Code formatted (Ruff) → ruff format --check
• Lint passes (Ruff) → ruff check .
• Types pass (mypy/pyright) → mypy src
• Tests pass (pytest) → pytest -q

• Backward compatibility considered
• Adequate comments for tricky parts
• CI green

Risk & Rollback Plan

Probably not needed in the beginning

Notes for Reviewers

[anagainaru]

Looks good