title	SignalOps
emoji	🏭
colorFrom	blue
colorTo	indigo
sdk	docker
app_port	7860
pinned	false

Nexusolve Manufacturing Copilot

Real-time manufacturing event ingestion with structured storage, vector search, and a rolling cache — all latency-instrumented.

Live demo (Hugging Face Space): runs the full stack — Postgres, Qdrant, Redis, FastAPI backend, React dashboard, and the event generator — in a single container (Dockerfile at repo root, started by entrypoint.sh on port 7860). For local development with the multi-service split, use docker compose up (see Cold start below).

Stack

• Backend: FastAPI (Python 3.11, fully async)
• Postgres: structured event store
• Qdrant: vector search over event text
• Redis: rolling state cache (checked before any Qdrant/Postgres read)
• Generator: synthetic event stream (singles, bursts, correlated chains)

Cold start (Docker Compose)

The whole system — Postgres, Qdrant, Redis, the FastAPI backend, the React dashboard, and the event generator — comes up from a cold start with one command. No API keys or accounts are required (the LLM agent runs in a grounded mock mode until you provide a key).

Prerequisites

• Docker Desktop running.
• On Windows: Docker Desktop uses the WSL2 backend — if docker info fails with a daemon/pipe error, install it once with wsl --install (elevated PowerShell), reboot, then start Docker Desktop. This is a host requirement, not a project one.
• Free ports: 8000 (backend), 5173 (dashboard), 6333 (Qdrant), 5432 (Postgres), 6379 (Redis).

Start everything

docker compose up --build

First run builds two Python images + the Node image and pulls Postgres/Qdrant/ Redis (a few minutes). On later runs, docker compose up -d (no --build) is near-instant. The backend waits for the datastores' healthchecks; the generator waits for the backend's /health; then events stream every 0.5–3s.

Verify it's up

docker compose ps                       # all services "Up"/"healthy"
curl http://localhost:8000/health       # {"status":"ok", ...}

Service	URL
Dashboard	http://localhost:5173
API docs (Swagger)	http://localhost:8000/docs
Health	http://localhost:8000/health
Qdrant UI	http://localhost:6333/dashboard

Optional — live LLM (instead of mock)

# bash
export ANTHROPIC_API_KEY=sk-ant-...
docker compose up --build

Or put ANTHROPIC_API_KEY=sk-ant-... in a .env file next to docker-compose.yml. Empty/unset → the agent uses the deterministic mock streamer (still grounded + cited). The key is never committed (.env is git-ignored).

Stop / resume / reset

docker compose stop      # pause (keeps data + images)
docker compose start     # resume
docker compose down      # remove containers (keeps named volumes)
docker compose down -v   # full reset (wipes Postgres/Qdrant/Redis data)

Architecture & design rationale, ranking weights, and full latency/load numbers: see docs/ARCHITECTURE.md.

Endpoints

• POST /ingest — accept one event; writes to Postgres + Qdrant + Redis in parallel (asyncio.gather), logs latency per stage, then publishes it to the live channel. Also updates rolling operational state (open issues, technicians).
• POST /ingest/batch — accept up to 500 events, ingested concurrently.
• GET /recommendations?top_n=5 — recommendation engine: ranks the top actions to take now (urgency × impact × technician_fit × recency), with reasoning + factor breakdown per item. Reads state from Redis only.
• GET /state — current rolling operational state (open issues + technician availability), Redis-only.
• GET /search?q=...&machine_id=&event_type=&time_window=48h&sla_breach= — hybrid search: BM25 (Postgres text) + semantic (Qdrant), fused 0.6·semantic + 0.4·bm25, with structured filters and per-path latency.
• POST /agent/explain — LLM explanation agent (SSE stream): explains why a recommendation ranks where it does, grounded in the event, its correlated chain, the ranking factors, and matching SOP excerpts. Every claim cites [event:..] / [machine:..] / [doc:..]. Logs TTFT, total latency, token usage; caches by event_state hash (identical state → cached, no LLM call).
• POST /agent/query — free-form query agent (SSE): answers supervisor questions grounded in hybrid search + live state + SOPs, with citations.
• GET /events/recent?limit=N — recent events, Redis-first then Postgres.
• GET /events/correlated/{correlation_id} — full event chain, Redis-first.
• WS /ws/events — live event feed (Redis pub/sub).
• WS /ws/recommendations — live ranked feed, pushed on every state change.
• GET /health — concurrent probe of all three stores.

Dashboard (http://localhost:5173)

React + Vite, live via WebSockets:

• Ranked recommendation feed (/ws/recommendations) with factor breakdown and a per-item "Why this? (LLM)" button that streams the cited explanation.
• Event timeline (/ws/events), scrollable history.
• Query interface — streams the agent's answer with citation highlighting.
• Latency panel — live recommendation_ms (vs the <100ms target) and retrieval_ms (visible proof of the performance work).

LLM latency engineering (Feature #5)

• claude-haiku-4-5 on the hot path — no extended thinking, no effort (errors on Haiku) — so first token is quick.
• Streaming — tokens forwarded as they arrive; time_to_first_token is logged separately from total_latency.
• Prompt caching — system prompt + SOP corpus marked cache_control: ephemeral.
• Application cache — identical event_state → cached explanation from Redis (measured ~0.3ms, zero tokens), vs a live generation.
• Token usage (input/output/cache) logged per call.

Runs with no API key: a deterministic mock streamer produces grounded, cited output so the full pipeline (streaming, caching, latency, dashboard) works offline. Set ANTHROPIC_API_KEY (shell or .env) and docker compose up to switch to live Haiku — no code change.

Measured latency (local, N=100/endpoint — see tests/benchmark.py)

Path	server p50	server p95	notes
Recommendations	~1.3ms	~2.0ms	redis_fetch + pure-CPU scoring; target <100ms
Hybrid search	~10ms	~16ms	semantic + BM25 run concurrently; per-path logged

Run the benchmark against the live stack:

pip install httpx
python tests/benchmark.py        # N=100 each; set N=200 / BACKEND_URL=... to vary

Under concurrent load (tests/loadtest.py, concurrency=25, 600 reqs)

Where measured	throughput	wall p95	server-handler p95	server max
Host → container (Windows Docker proxy)	~180 req/s	~400 ms	3.4 ms	40 ms
Container → container (in Docker network)	~440 req/s	~160 ms	3.2 ms	8.9 ms

The recommendation handler stays ~3ms p95 even at 25-way concurrency — it is not the bottleneck. Wall-clock tail latency on Windows is dominated by Docker Desktop's host→WSL2 port-forwarding proxy (the ~240ms gap between the two rows above), which a Linux host / hosted deployment does not pay. Always read the server-measured latency_ms the endpoints return — that is the real engine cost; wall-clock over localhost on Windows is inflated by the proxy.

The backend runs WEB_CONCURRENCY (default 4) uvicorn workers so concurrent requests aren't serialised on one event loop; all state is in Redis/Postgres/Qdrant, so workers are stateless and safe to scale.

Scaling the recommendation engine (measured optimisation)

A small demo (~20 open issues) hides scaling cost. Injecting synthetic open issues (tests/scaletest.py) and measuring server-side total p50 latency:

Open issues	v0: Pydantic-per-event O(N)	v1: raw-dict scoring O(N)	v2: sorted-set top-K O(K)
~500	12.6 ms	6.8 ms	1.2 ms
~1,000	24.0 ms	12.2 ms	1.1 ms
~2,000	48.1 ms	23.3 ms	1.2 ms
~3,000	71.8 ms	34.8 ms	1.3 ms
~6,000	—	—	1.8 ms

• v0 → v1: stop re-validating a Pydantic model per open event (events are validated once at ingest); score directly off the Redis dicts and build models only for the final top-N. ~5× faster scoring; the json.loads of every blob then dominates.
• v1 → v2: store open issues in a Redis sorted set scored by static base priority (urgency × impact, computed at ingest). The hot path ZREVRANGEs only the top candidate_window (50) candidates and fully re-scores those with live recency + technician-fit. Latency goes flat (~1–2 ms) regardless of total open-issue count — O(log N + K) instead of O(N).

Tradeoff (candidate window): v2 re-ranks only the top-50 by static priority. Because the open-issue TTL (30 min) ≈ the recency half-life, recency stays ≥ ~0.37 within the window, so urgency dominates and the true top-5 are reliably inside the top-50. Widen candidate_window to trade a little latency for a larger safety margin; set it to cover all issues to disable the approximation.

Where latency is spent

• Recommendations: dominated by the single Redis fetch (top-K + technicians + count, one gather); scoring is ~0.2 ms. The hot path never touches Postgres or Qdrant. Technician availability is served from a short-TTL (2s) in-process cache to shave a Redis round-trip per call, trading bounded staleness for lower cost.
• Hybrid search: dominated by the two retrieval paths (semantic ≈ BM25), which run concurrently via asyncio.gather; fusion is <1ms. BM25 scoring runs in a thread (asyncio.to_thread) to keep the event loop free.

Embeddings use fastembed (BAAI/bge-small-en-v1.5, 384-dim, CPU), run via asyncio.to_thread and baked into the image at build time. Real semantic scores (e.g. 1.0 / 0.95 / 0.85 for a relevant query) add only single-digit ms to the search path; the recommendation engine doesn't embed and is unaffected.

Tests

Schema tests are pure Python and need no running services:

pip install -r tests/requirements.txt
pytest tests/test_models.py

Latency instrumentation

Every store-touching function emits a JSON log line with stage, latency_ms, and event_id. See backend/logging_config.py (stage_timer, track_latency).

Notes

• backend/db.py::embed uses fastembed (BAAI/bge-small-en-v1.5, CPU, baked into the image) for real local semantic embeddings — no external API, no egress.

Layout

/backend     FastAPI app (main, models, ingestion, db, logging_config, config)
/generator   Event stream generator
/frontend    React dashboard (to be added)
/docs        SOP documents (to be added)
/tests       Load + unit tests (to be added)