feat: fabric stability engine + LLM-as-Judge playbook scoring (v0.4.0)

[gesh75]

Why

Builds the real, verified version of the "Phase 12" feature set that previously existed only as text in an unexported chat session — implemented natively on current main so it composes with the streaming pipeline, sanitize gate, and template miner already there.

📶 Fabric stability engine (stability.py)

Classified events tell you what broke; this tells you what keeps breaking:

• Flap detection — down↔up oscillation of the same entity class per device (interfaces, BGP peers, OSPF neighbors, BFD sessions, LAG members, VPN tunnels). Three bounces score far worse than one clean failure, which a severity-ranked list can't express.
• Volume bursts — peak-minute event rate vs the device's own median minute.
• Trend + 24h risk band — rising/stable/falling activity across the window, projected into a deterministic low/medium/high risk band.
• Wired into analyze()'s single streaming pass: O(devices × entity classes) memory, minute-prefix bucketing with arrival-order fallback (network syslog timestamps are unreliable and sometimes absent).
• Surfaces as stability in /api/analyze + MCP analyze_logs, and as a Fabric Stability UI panel (worst devices first, plain-language recommendations).

⚖️ LLM-as-Judge playbook scoring (judge.py + ai-log-analyzer eval)

The product's core output is LLM/KB-generated remediation playbooks — nothing measured their quality until now:

• Scores 0–10 on actionability (real runnable CLI, expected outputs, preventive config), safety (disruptive commands need context; empty playbooks can't ace safety "by absence"), grounding (real device names in, R1/SW2-style placeholders out), completeness (root cause, risk, verify phase, monitoring rules with numeric thresholds).
• Deterministic heuristic core (zero deps, CI-safe); --use-llm blends a real LLM judge through the configured provider chain with silent fallback.
• ai-log-analyzer eval self-tests all 41 actionable KB playbooks by default (currently 9.8/10 overall — the quality floor with the LLM off), scores a saved /api/analyze result with --file, and --min-score N turns it into a CI quality gate.

Version

0.3.1 → 0.4.0 (this plus the merged PR #12 feature set makes an honest minor release; the CHANGELOG Unreleased section is now the 0.4.0 entry).

Tests

325 → 338 passed (2 skipped), ruff clean. New coverage: flap/burst/trend/risk scoring, bounded-memory guarantee, analyze integration, judge scoring semantics (good vs empty, disruptive-command penalty, placeholder penalty), LLM-blend fallback + merge, and both eval CLI paths including the CI gate.

🤖 Generated with Claude Code

https://claude.ai/code/session_015XM7j9iDXYwSvjnxZYjzaq

---

Generated by Claude Code