gesh75 · gesh75 · Jul 13, 2026 · Jul 13, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,10 +4,29 @@ All notable changes to **netlog-ai** are documented here.
 Format follows [Keep a Changelog](https://keepachangelog.com/); this project uses
 loose semantic versioning.
 
-## [Unreleased]
+## [0.4.0] - 2026-07-13
 
 ### Added
 
+- **Fabric stability engine** (`stability.py`) — per-device flap detection
+  (down↔up oscillation of interfaces / BGP peers / OSPF neighbors / LAG
+  members / VPN tunnels), event-rate burst detection against each device's
+  own baseline, rising/stable/falling trend, and a deterministic 24h risk
+  band. Surfaces as `stability` in `/api/analyze` + MCP and a **📶 Fabric
+  Stability** panel in the UI (worst devices first, actionable
+  recommendations). Streaming-safe: O(devices × entity classes) memory,
+  minute-prefix bucketing with arrival-order fallback so unreliable syslog
+  timestamps can't break it.
+- **LLM-as-Judge playbook scoring** (`judge.py` + `ai-log-analyzer eval`) —
+  scores generated playbooks 0–10 on actionability / safety / grounding /
+  completeness with a deterministic heuristic core (empty playbooks can't
+  ace safety "by absence"; disruptive commands without context are
+  penalized; R1/SW2-style placeholder names hurt grounding). Optional
+  `--use-llm` blends a real LLM judge via the configured provider, falling
+  back silently. `eval` self-tests every rule-based KB playbook by default,
+  scores a saved analyze result with `--file`, and `--min-score` makes it a
+  CI quality gate.
+
 - **Unknown-pattern template mining** (`patterns.py`) — every line the regex KB
   can't match is mined into templates with a dependency-free Drain-style
   streaming clusterer (variables masked as `<ip>/<mac>/<hex>/<if>/<n>/<ts>`,

diff --git a/README.md b/README.md
@@ -15,7 +15,7 @@
 
 > **Network logs in. Ranked actions out.** A local, dark-themed dashboard that classifies syslog events from any vendor (Junos, Arista EOS, FRR), builds a prioritized action list, and lets an LLM write the root-cause analysis with copy-pastable CLI fixes.
 
-[![CI](https://github.com/gesh75/netlog-ai/actions/workflows/ci.yml/badge.svg)](https://github.com/gesh75/netlog-ai/actions/workflows/ci.yml) ![Tests](https://img.shields.io/badge/tests-325%20passing-brightgreen) ![License](https://img.shields.io/badge/license-MIT-blue) ![Python](https://img.shields.io/badge/python-3.10%2B-blue) ![Stack](https://img.shields.io/badge/stack-Flask%20%2B%20vanilla%20JS-1f6feb)
+[![CI](https://github.com/gesh75/netlog-ai/actions/workflows/ci.yml/badge.svg)](https://github.com/gesh75/netlog-ai/actions/workflows/ci.yml) ![Tests](https://img.shields.io/badge/tests-338%20passing-brightgreen) ![License](https://img.shields.io/badge/license-MIT-blue) ![Python](https://img.shields.io/badge/python-3.10%2B-blue) ![Stack](https://img.shields.io/badge/stack-Flask%20%2B%20vanilla%20JS-1f6feb)
 
 > 📓 Recent changes — cross-source correlation + per-device triage (MCP tools **and** Device-tab UI) — are in [`CHANGELOG.md`](CHANGELOG.md).
 
@@ -69,6 +69,8 @@ Tools exposed: `list_sources`, `add_source`, `fetch_logs`, `search_logs`,
 | 🔎 **Classify** | 50+ regex patterns across Junos, EOS, FRR, IOS, RFC-3164/5424 — plus your own rules via `AI_LOG_ANALYZER_CUSTOM_RULES` / `POST /api/rules` |
 | 🔭 **Unknown patterns** | Lines no rule matched are template-mined (Drain-style, zero deps): variables masked, shapes clustered, error-smelling templates ranked first — novel failure modes surface instead of vanishing as `info` noise. Set `AI_LOG_ANALYZER_TEMPLATE_STORE` for cross-run memory: shapes never seen in any prior run get flagged 🆕 |
 | 🧭 **Prioritize** | Deduped action items, ranked by severity × count, recovery events excluded |
+| 📶 **Fabric stability** | Per-device flap detection (interface/BGP/OSPF/LAG/VPN down↔up oscillation), event-rate bursts vs the device's own baseline, trend + heuristic 24h risk band — in `/api/analyze` and its own UI panel |
+| ⚖️ **LLM-as-Judge** | `ai-log-analyzer eval` scores playbooks 0–10 on actionability/safety/grounding/completeness — heuristic core, optional real LLM judge, `--min-score` CI gate |
 | 🧠 **Deep analyze** | Top-N items get an LLM-written root-cause + risk + remediation playbook |
 | 🛡️ **Sanitize-first** | Every config/log payload is scrubbed (`$6$`, `$9$`, SSH keys, SNMP, RADIUS, public IPs) before LLM call |
 | 📈 **Health score** | Weighted formula → 0–100 + A/B/C/D/F + sparkline trend |
@@ -106,6 +108,11 @@ ai-log-analyzer analyze --frr r1 r2 --no-llm | jq .score
 ai-log-analyzer analyze --file /var/log/syslog
 docker logs my-router 2>&1 | ai-log-analyzer analyze --stdin
 
+# Score playbook quality (LLM-as-Judge; KB self-test by default)
+ai-log-analyzer eval                 # heuristic, offline
+ai-log-analyzer eval --use-llm       # blend a real LLM judge
+ai-log-analyzer eval --file result.json --min-score 7   # CI gate
+
 # Run the full test suite
 pytest --cov=src --cov-report=term-missing
 ```
@@ -319,6 +326,8 @@ flowchart TB
 src/ai_log_analyzer/
   classifier.py        50+ regex patterns + severity/category lookup + custom rules
   patterns.py          Drain-lite template miner for lines no rule matched
+  stability.py         Flap/burst/trend scoring + 24h risk per device
+  judge.py             LLM-as-Judge playbook quality scoring (eval CLI)
   kb.py                Rule-based deep-analysis KB (fallback when LLM is off)
   llm.py               Docker Model Runner (TCP + UDS) + Anthropic Claude
   analyzer.py          End-to-end pipeline: classify → actions → score → summary
@@ -398,7 +407,7 @@ src/ai_log_analyzer/
 
 ## Tested & accessible
 
-- **325 unit + integration tests** (pytest)
+- **338 unit + integration tests** (pytest)
 - Frontend audited across 8 review rounds:
   - WCAG-AA: `:focus-visible` rings, `aria-live` regions, `role=tablist/tab/tabpanel`, skip-to-main link, `prefers-reduced-motion` fallback
   - Responsive ≤ 1100px, PWA-ready (`theme-color`, `mobile-web-app-capable`, SVG favicon)
@@ -416,7 +425,8 @@ src/ai_log_analyzer/
 - [x] Custom classification rules — JSON file (`AI_LOG_ANALYZER_CUSTOM_RULES`) + runtime `POST /api/rules`; UI editor still open
 - [x] Unknown-pattern template mining — Drain-style clustering of unmatched lines, surfaced in UI/API/MCP
 - [x] Cross-run template persistence — `AI_LOG_ANALYZER_TEMPLATE_STORE` flags never-before-seen shapes (🆕)
-- [ ] Per-template volume anomaly detection (sudden rate spike/drop per device)
+- [x] Fabric stability scoring — flap/burst/trend detection + 24h risk band per device
+- [x] LLM-as-Judge playbook quality scoring (`ai-log-analyzer eval`, CI-gate ready)
 
 ## Contributing
 

diff --git a/pyproject.toml b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "netlog-ai"
-version = "0.3.1"
+version = "0.4.0"
 description = "AI-powered network syslog analyzer with pluggable LLM providers (local Docker Model Runner / Anthropic Claude) and lab adapters."
 readme = "README.md"
 license = { text = "MIT" }

diff --git a/src/ai_log_analyzer/analyzer.py b/src/ai_log_analyzer/analyzer.py
@@ -17,6 +17,7 @@
 )
 from ai_log_analyzer.patterns import TemplateMiner, apply_template_store
 from ai_log_analyzer.sanitize import sanitize
+from ai_log_analyzer.stability import StabilityTracker
 
 
 # Recovery events are classified as medium because they're useful in the
@@ -91,6 +92,7 @@ class AnalysisResult:
     llm_powered: bool
     generated_at: str
     unknown_patterns: dict = field(default_factory=dict)
+    stability: dict = field(default_factory=dict)
 
     def to_dict(self) -> dict:
         return {
@@ -106,6 +108,7 @@ def to_dict(self) -> dict:
             "llm_powered": self.llm_powered,
             "generated_at": self.generated_at,
             "unknown_patterns": self.unknown_patterns,
+            "stability": self.stability,
         }
 
 
@@ -515,7 +518,7 @@ def _executive_summary(
 
 def _aggregate_stream(
     classified: Iterable[ClassifiedEvent], top_k: int = 300,
-) -> tuple[list[ClassifiedEvent], dict[str, int], dict[str, int], dict, dict, TemplateMiner]:
+) -> tuple[list[ClassifiedEvent], dict[str, int], dict[str, int], dict, dict, TemplateMiner, StabilityTracker]:
     """Single bounded pass over a classified stream.
 
     Memory is O(top_k + KB rules + devices + template clusters) regardless of
@@ -532,12 +535,14 @@ def _aggregate_stream(
     groups: dict[tuple[str, str], dict] = {}
     by_host: dict[str, dict] = {}
     miner = TemplateMiner()
+    tracker = StabilityTracker()
 
     for seq, e in enumerate(classified):
         sev_counts[e.severity] = sev_counts.get(e.severity, 0) + 1
         cat_counts[e.category] = cat_counts.get(e.category, 0) + 1
         _group_actionable(e, groups, seq)
         _count_device(e, by_host)
+        tracker.add(e)
         if e.category == "other" and e.message:
             miner.add(e.message, e.hostname)
         heap = buckets.setdefault(e.severity, [])
@@ -554,7 +559,7 @@ def _aggregate_stream(
         )
         if len(top_events) >= top_k:
             break
-    return top_events[:top_k], sev_counts, cat_counts, groups, by_host, miner
+    return top_events[:top_k], sev_counts, cat_counts, groups, by_host, miner, tracker
 
 
 def analyze(events: Iterable[LogEvent], use_llm: bool = True, llm_top_n: int = 3) -> AnalysisResult:
@@ -565,7 +570,7 @@ def analyze(events: Iterable[LogEvent], use_llm: bool = True, llm_top_n: int = 3
     event list. Thread-safe: never mutates global llm state. `use_llm=False`
     is enforced by forcing `llm_top_n=0` and disabling the LLM exec summary.
     """
-    top_events, sev_counts, cat_counts, groups, by_host, miner = _aggregate_stream(
+    top_events, sev_counts, cat_counts, groups, by_host, miner, tracker = _aggregate_stream(
         iter_classify(events)
     )
     # Opt-in cross-run persistence: flags templates never seen in ANY prior
@@ -595,6 +600,7 @@ def analyze(events: Iterable[LogEvent], use_llm: bool = True, llm_top_n: int = 3
         llm_powered=any_llm,
         generated_at=datetime.now(timezone.utc).isoformat(timespec="seconds"),
         unknown_patterns=miner.to_dict(top_n=10),
+        stability=tracker.report(),
     )
 
 

diff --git a/src/ai_log_analyzer/cli.py b/src/ai_log_analyzer/cli.py
@@ -48,6 +48,48 @@ def cmd_containers(_args: argparse.Namespace) -> int:
     return 0
 
 
+def cmd_eval(args: argparse.Namespace) -> int:
+    """Score playbook quality with the LLM-as-Judge harness.
+
+    Default: self-test every rule-based KB playbook (the quality floor with
+    the LLM off). --file scores a saved /api/analyze JSON result instead.
+    --min-score turns the run into a CI gate (exit 1 below threshold).
+    """
+    from ai_log_analyzer import judge
+
+    if args.file:
+        try:
+            with open(args.file, encoding="utf-8") as fh:
+                result = json.load(fh)
+        except (OSError, json.JSONDecodeError) as exc:
+            print(f"error: cannot read {args.file}: {exc}", file=sys.stderr)
+            return 2
+        report = judge.judge_analysis_result(result, use_llm=args.use_llm)
+    else:
+        report = judge.judge_kb(use_llm=args.use_llm)
+
+    if args.json:
+        print(json.dumps(report, indent=2))
+    else:
+        print(f"Playbooks scored : {report['playbooks_scored']}")
+        print(f"Overall quality  : {report['overall']}/10")
+        print()
+        for v in report["verdicts"][: args.show]:
+            s = v["scores"]
+            label = v.get("description") or v.get("category", "")
+            print(f"  {s['overall']:>4}/10  [{v.get('judge')}] {label[:70]}")
+            print(f"          action={s['actionability']} safety={s['safety']} "
+                  f"grounding={s['grounding']} complete={s['completeness']}")
+            for note in v.get("notes", [])[:2]:
+                print(f"          ! {note}")
+
+    if args.min_score and report["overall"] < args.min_score:
+        print(f"\nFAIL: overall {report['overall']} < required {args.min_score}",
+              file=sys.stderr)
+        return 1
+    return 0
+
+
 def cmd_mcp(args: argparse.Namespace) -> int:
     """Start the MCP server (stdio by default) so Claude Code / Cursor / Continue
     can call netlog-ai tools directly."""
@@ -83,6 +125,14 @@ def main() -> None:
     cp = sub.add_parser("containers", help="List running FRR lab containers")
     cp.set_defaults(func=cmd_containers)
 
+    ep = sub.add_parser("eval", help="Score playbook quality (LLM-as-Judge; KB self-test by default)")
+    ep.add_argument("--file", help="Saved /api/analyze JSON result to score (default: score the built-in KB)")
+    ep.add_argument("--use-llm", action="store_true", help="Blend in a real LLM judge (falls back to heuristic)")
+    ep.add_argument("--json", action="store_true", help="Machine-readable output")
+    ep.add_argument("--show", type=int, default=10, help="Verdicts to print, worst first (default 10)")
+    ep.add_argument("--min-score", type=float, default=0.0, help="Exit 1 if overall quality is below this (CI gate)")
+    ep.set_defaults(func=cmd_eval)
+
     mp = sub.add_parser("mcp", help="Run as an MCP server (for Claude Code / Cursor / Continue)")
     mp.add_argument("--transport", default="stdio", choices=["stdio", "streamable-http"],
                     help="MCP transport (default: stdio)")