logdrain

A streaming log pattern-miner. Stream raw log lines in; logdrain clusters them into templates in real time with the Drain algorithm, flags per-template rate anomalies with an online EWMA, and pushes both out over Server-Sent Events — with no external service and no LLM.

"Accepted connection from 10.0.0.7 port 22"  ┐
"Accepted connection from 10.0.0.9 port 22"  ┼─▶  Accepted connection from <IP> port <NUM>   (×523)
"Accepted connection from 10.0.1.2 port 22"  ┘

Live demo: https://logdrain.lucashvieira.dev — it seeds itself on boot, so it's instantly alive. Or run it locally in a minute, see Run it locally.

The interesting part: a from-scratch Drain parser

drain.py implements Drain (He et al., "Drain: An Online Log Parsing Approach with Fixed Depth Tree", ICWS 2017) — the real, published algorithm behind modern log analytics, not a regex bag or an LLM wrapper.

A line is routed through a fixed-depth tree — first by its token count, then by its leading tokens — to a small set of candidate clusters. At the leaf, the most similar cluster (by token-position overlap above a threshold) absorbs the line, generalising any differing positions to a <*> wildcard; if nothing matches, a new cluster is born. Every step is O(depth) with no scan of the corpus. A deterministic masking pass collapses timestamps, IPs, UUIDs, numbers, and paths to placeholders first, so templates stay readable and the tree doesn't explode on high-cardinality fields.

On top of it, an online EWMA detector tracks each template's per-minute rate in constant memory and flags counts that deviate from the baseline by more than z standard deviations.

A free, always-green demo

Everything runs against Postgres with zero third-party API cost, so the demo never incurs usage charges and stays up. Hitting POST /v1/seed (or booting with LOGDRAIN_SEED_ON_START=true) replays a bundled corpus so you immediately see templates form and a rate anomaly fire over the live SSE feed.

How correctness is proven

The parser is the kind of thing that's easy to get subtly wrong, so it's pinned down:

• Invariants — identical lines collapse to one template; lines differing only by a number share a template (via masking); a trailing-token difference generalises to a wildcard; structurally different lines stay separate.
• A golden snapshot (test_golden.py) asserts the bundled corpus always mines exactly the same set of six templates, so a regression in masking or the parser can't silently change the demo's output.
• Anomaly detection is tested on synthetic series (a noisy baseline plus a spike), and the full ingest→persist→anomaly path is tested against real Postgres (Testcontainers in CI).

The whole codebase is mypy --strict clean and Ruff-linted.

Tech stack

• FastAPI (async) + Pydantic v2 + sse-starlette
• SQLAlchemy 2.0 async + asyncpg over PostgreSQL
• pure, fully-typed Drain / masking / EWMA modules (no framework imports)
• pytest + Testcontainers, Ruff, mypy --strict, managed with uv

Run it locally

uv sync
createdb logdrain
export LOGDRAIN_DATABASE_URL="postgresql+asyncpg://localhost:5432/logdrain"

uv run uvicorn logdrain.app:app --reload

Open http://localhost:8000, click Replay sample corpus, and watch templates appear and an anomaly fire in the live feed. The interactive OpenAPI docs are at /docs.

Or with Docker (self-seeding demo):

docker compose up --build      # http://localhost:8000

Run the checks (tests use Testcontainers, or set LOGDRAIN_TEST_PG):

uv run pytest        # uv run ruff check .   ·   uv run mypy src

API

Method	Path	Description
POST	/v1/ingest	ingest log lines; returns the matched/created template per line
GET	/v1/templates	discovered templates with counts and examples
GET	/v1/anomalies	rate anomalies detected so far
GET	/v1/stream	SSE feed of new_template and rate_anomaly events
POST	/v1/seed	replay the bundled sample corpus
GET	/health	liveness

Data model

templates (id, cluster_id unique, template, token_count, count, example_line, first_seen, last_seen)
anomalies (id, cluster_id, kind, z_score, observed, baseline, detected_at)

Limitations

Drain assumes the leading tokens of a line are stable (the event type), with variable params later or numeric — true for the vast majority of application logs. Templates and rate state live in the app process and are rebuilt from the stream; persistence mirrors them for query.

License

MIT