feat: unknown-pattern template mining + custom rules; close sanitize/auth gaps

[gesh75]

Why

A deep product-analysis pass: market research on how production AIOps pipelines handle network syslog (Drain3 / IBM DeCorus / IncidentFox-class tools), plus a full audit of the repo's development history (PRs #1–#11). The single biggest analytical gap: any log line the 55-pattern regex KB can't match silently becomes info/other and disappears — while the strongest early-warning signal in the field is precisely "a message shape never seen before just appeared."

Added

• 🔭 Unknown-pattern template mining (patterns.py, zero dependencies) — every unmatched line is masked (<ip>/<mac>/<hex>/<if>/<n>/<ts>) and clustered with a Drain-style fixed-depth miner (similar shapes merge into <*> wildcards, LRU-bounded memory so the streaming pipeline stays flat at any input size). Results surface as unknown_patterns in /api/analyze, the MCP analyze_logs tool, and a new Unknown Patterns UI panel — error-smelling templates ranked first.
• 🆕 Cross-run template persistence — set AI_LOG_ANALYZER_TEMPLATE_STORE and the miner remembers every shape across runs (FIFO-bounded JSON store, atomic writes, corrupt-file tolerant). Shapes never seen in any prior run are flagged is_new / new_template_count / 🆕 in the UI — the classic AIOps early-warning.
• Custom classification rules (roadmap item) — a JSON rules file loaded at startup (AI_LOG_ANALYZER_CUSTOM_RULES) plus a runtime API (GET/POST /api/rules). Custom rules run before the built-in KB, so operators can demote known noise or promote site-specific failures.

Security (audit findings)

• Log→LLM path now passes the sanitize gate. deep_analyze() sent raw sample log lines (and severity-promoted raw descriptions) to the LLM unsanitized; config paths were always scrubbed but logs were not — contradicting the project's core sanitize-before-LLM guarantee.
• /api/sources/<id>/test and /fetch now require the API token — the only POST data routes missing the gate (data egress + probe surface).
• Authorization: Bearer accepted alongside X-API-Token — the README documented Bearer while the code only accepted the custom header, locking out users who followed the docs.

Fixed

• MCP server resolves packaged data/sites — list_sites/analyze_site returned empty on any pip install netlog-ai[mcp] wheel (repo-root-only lookup).
• /api/optimize/site now uses the shared site loader — per-device platform preserved on mixed-vendor sites (was stamping every device with e.g. multi (Nokia SRL + Arista cEOS + FRR)).
• Syslog connector fetch() is idempotent — it drained the ring buffer, so correlate → triage on the same source saw zero events on the second call.
• Default Ollama model gemma4:latest → qwen2.5-coder:latest (gemma4 doesn't exist and ollama is the default provider — out-of-box LLM calls failed silently).
• tfsm_fire end-to-end tests skip gracefully when the upstream template-DB download 404s (upstream restructured); TFSM_DB_URL override added for mirrors.
• Env source loader no longer leaks verify_tls/timeout_seconds into extra.
• .env.example/README now document the real security env vars (AI_LOG_ANALYZER_API_TOKEN, AI_LOG_ANALYZER_CORS_ORIGINS, ANALYZER_HOST, AI_LOG_ANALYZER_FILE_ROOTS), the ollama provider, and correct defaults (LLM_TIMEOUT=120).

Tests

294 → 325 passed (2 skipped), ruff clean. New coverage: miner masking/clustering/eviction/integration, template-store persistence, custom rules (engine + API + precedence), sanitize-gate regressions, auth regressions, syslog idempotency, plus a coverage wave for the modules earlier phases left untested — reports.py, runbook.py, topology.py, and web routes /api/report, /api/runbook, /api/topology, /api/diff, /api/copilot.

Suggested next features (researched, not in this PR)

1. Per-template volume anomaly detection — count templates per device per time window; sudden rate spikes/drops flag outages before severity rules do.
2. Incident timeline reconstruction — burst-window grouping across devices to identify the first trigger event in a cascade.
3. Real-time tail mode (existing roadmap) — the syslog connector + miner are now both streaming-safe, making a websocket live view straightforward.
4. Vendor the tfsm template DB or pin a mirror — upstream is gone, so netlog-ai[parse] has no DB source for fresh installs.

🤖 Generated with Claude Code

https://claude.ai/code/session_015XM7j9iDXYwSvjnxZYjzaq