Add semantic drift monitor with CLI and security hardening

[Knapp-Kevin]

This PR introduces an optional Semantic Drift Monitor for Surveyor, along with a focused round of security and robustness hardening.

The drift monitor adds temporal awareness to Surveyor’s clustering output by comparing cluster structure across runs. It does not alter clustering behavior or writeback logic. It is fully optional and disabled by default.

The security fixes address availability, path confinement, atomic writes, retention, and config validation.

---

What’s Included

1. Semantic Drift Monitor (Optional)

New module:

• drift.py
  • Persists cluster snapshots to JSON
  • Compares latest two snapshots
  • Computes:
    • Jaccard similarity
    • Membership churn
    • New clusters
    • Dissolved clusters
    • Split/merge heuristics
  • Writes drift reports to JSON payloads in .log files

Surveyor integration:

• Drift runs after clustering and labeling
• No changes to clustering logic
• No changes to writeback behavior
• Fully non-blocking
• Wrapped in try/except to prevent pipeline failure

CLI additions:

• alfred drift status
• alfred drift history [--limit N]
• alfred drift show <timestamp|filename>

Configuration:

features:
  semantic_drift_monitor: false

semantic_drift:
  snapshot_retention: 10
  similarity_threshold: 0.5
  warn_on_high_drift: true

[Copilot]

Pull request overview

Adds an optional “Semantic Drift Monitor” to Surveyor to persist cluster snapshots, compare consecutive runs, and expose drift summaries via new CLI commands, along with config + documentation updates.

Changes:

• Introduce DriftMonitor to snapshot cluster membership and write drift reports.
• Integrate drift processing into the Surveyor daemon and add alfred drift {status|history|show} CLI.
• Add config schema + examples/docs for feature flag and drift settings.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/alfred/surveyor/drift.py	New snapshot + drift comparison logic, report writing, pruning, and path confinement for report reads
src/alfred/surveyor/drift_cli.py	New CLI handlers for drift status/history/show
src/alfred/surveyor/daemon.py	Integrates drift processing into the Surveyor pipeline after clustering/labeling
src/alfred/surveyor/config.py	Adds FeatureFlags and SemanticDriftConfig to typed config loading
src/alfred/cli.py	Adds top-level alfred drift command wiring and dispatch
src/alfred/quickstart.py	Includes drift feature/config defaults in generated config
src/alfred/_bundled/config.yaml.example	Documents new config keys in bundled example
config.yaml.example	Documents new config keys in root example
docs/Surveyor.md	Documents drift settings and artifacts
docs/CLI-Commands.md	Documents new alfred drift commands
INITIAL_SECURITY_REVIEW.md	Adds security review notes for drift monitor

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The control flow now logs daemon.no_changed_clusters but still continues into drift processing and then logs daemon.labeling_complete. This results in two potentially contradictory log events for the no-change case. Consider returning early after no_changed_clusters, or renaming/guarding the later log so it only fires when labeling actually ran (or rename it to reflect the full step including drift).

[Copilot]

Imports inside the per-cluster loop (from .state import ClusterState, from datetime import ...) reduce readability and are executed repeatedly (even if cached). Since ClusterState and datetime/timezone are already safe to import at module scope, prefer moving these imports to the top of the file (or at least above the loop).

[Copilot]

The docs say “All configuration lives in config.yaml under the surveyor section.” but the example immediately places features: and semantic_drift: at the top level (and the loader reads them from the top level). Update the wording (or the example structure) so users aren’t misled about where these keys must live.

[Copilot]

This sentence is inaccurate: the alfred drift ... commands are still available when the feature flag is disabled; they just print a message and exit. Suggest rephrasing to clarify that drift artifacts/reports are only generated when enabled, and the CLI will report “disabled” otherwise.

Suggested change

	`alfred drift ...` commands are available only when `features.semantic_drift_monitor` is enabled.
	`alfred drift ...` commands are always available, but drift artifacts/reports are only generated when `features.semantic_drift_monitor` is enabled; otherwise, the CLI reports that semantic drift monitoring is disabled and exits.

[Copilot]

cmd_status indexes required fields on latest using latest[...] (e.g., previous_cluster_count). If a drift report JSON is tampered with or partially written but still valid JSON, this will raise KeyError and crash the CLI. Prefer latest.get(...) with sensible defaults (and/or validate required keys after loading) so alfred drift status remains robust.

[Copilot]

cmd_drift doesn’t call _setup_logging_from_config(raw), so structlog warnings from drift artifact parsing (and any other surveyor logging used by the drift CLI) may be missing or inconsistently formatted compared to other commands (e.g. alfred surveyor, alfred temporal). Consider initializing logging the same way here.