realtime-ml-pipeline

Streaming fraud scoring, end to end, a durable event log with consumer groups and at-least-once delivery (crash-recovery is proven, not claimed), online windowed feature computation, real-time model inference, and PSI drift alerting. The architecture of a production streaming scorer, with every guarantee demonstrated by a measurement.

pipeline-demo                 # produce → score → measure all four properties
pipeline-demo --events 10000 --json

Architecture

Interactive/exportable version: docs/assets/architecture.html.

Architecture

producer (labeled synthetic transactions, fraud bursts injected)
   │ append (durable JSONL log, monotonic offsets)
   ▼
Broker — consumer groups, PENDING ledger, redelivery        pipeline/broker.py
   │ claim(batch) ──────────────► consumer crashes? unacked events
   ▼                              go stale → REDELIVERED to a peer
Consumer loop                                              pipeline/consumer.py
   ├─ OnlineFeatures: per-card sliding window               pipeline/features.py
   │    amount_over_mean · txn velocity · foreign-country
   ├─ model.score (LogReg trained on a historical replay    pipeline/model.py
   │    THROUGH THE SAME feature code — train/serve consistent)
   ├─ DriftMonitor: rolling PSI vs training distribution    pipeline/monitor.py
   └─ ACK (after processing — this ordering IS at-least-once)

The broker implements the exact Redis Streams contract (XADD/XREADGROUP/XACK/XAUTOCLAIM) on a local append-only log, so the delivery semantics are testable in-process, and swapping in real Redis is one adapter with the same five methods.

Measured results

pipeline-demo, 3000 events:

property	result
throughput	~10,000 events/s (single consumer, in-process)
per-event latency	p50 0.044 ms · p99 0.069 ms
fraud detection	precision 82% · recall 92% vs injected ground truth
false drift alarms (normal stream)	0
drift alarms (drifted stream)	3 alerts, PSI 2.88 vs 0.2 threshold
crash recovery	consumer killed after 150 events → rescue consumer redelivered and processed the remaining 650, 0 of 800 events lost, pending ledger empty

The two guarantees, and how they're proven

• At-least-once delivery. The consumer ACKs after processing. Kill it mid-batch and its claimed-but-unacked events sit in the group's PENDING ledger until a peer calls redeliver and picks them up, the demo and test_end_to_end_at_least_once_no_loss both count exactly zero lost events. The price of at-least-once is duplicates on redelivery; the consumer dedups by offset (idempotent processing), and the demo counts those too.
• Train/serve feature consistency. The model trains on a historical replay pushed through the same OnlineFeatures class that serves live traffic. Computing training features in a batch job and serving features in stream code is how streaming scorers silently rot, one code path removes the class of bug.

Why windowed features (not raw events)

A $600 transaction means nothing in isolation; $600 on a card whose rolling mean is $80, third transaction in 30 seconds, from a country the card has never used is the actual fraud signature. amount_over_mean + velocity + foreign-flag is what lifts detection to 82%/92%, raw amount alone can't separate a whale customer from a compromised card.

Install & test

pip install -e ".[dev]"
pytest -q          # 7 passed — broker ack/pending semantics, durability across reload,
                   # redelivery of stale claims, window aging, zero-loss crash recovery,
                   # detection >> base rate, drift fires on shift only

Pure Python + NumPy + scikit-learn, no external services; CI runs everything.

License

MIT