DeckStream

Real-time DJ recommendation dashboard — load a track on deck 1 in Serato, instantly see the 10 best harmonically compatible tracks from your library.

How it works

Serato DJ Pro (deck 1)
    → serato_event_agent.py  (Windows host, ReadDirectoryChangesW)
    → Kafka topic: serato.track.events  (Redpanda broker)
    → Spark Structured Streaming  (Delta Lake lookup + Camelot scoring)
    → Kafka topic: serato.recommendations
    → SSE → browser  (sub-second latency)

1. serato_event_agent.py watches master.sqlite on the Windows host. Only deck 1 triggers the pipeline — deck 2 (preparation) is ignored.
2. Spark scores recommendations by harmonic key (Camelot Wheel), BPM, folder proximity, and acoustic similarity.
3. Dashboard at http://localhost:8050 updates instantly via Server-Sent Events.

---

Requirements

• Docker Desktop (Windows, with WSL2 backend)
• Python 3.10+ on the Windows host (for serato_event_agent.py)
• Serato DJ Pro 4+
• NVIDIA GPU (optional — only needed for acoustic embeddings)

---

Quick start

1. Start the Docker stack

# First run — build images (takes a few minutes)
docker compose build

# Start all services
docker compose up -d

# Check everything is up
docker compose ps

2. Start the Windows host agent

Open a separate PowerShell terminal:

# Install dependencies once
pip install watchdog kafka-python

# Start the agent (keep this terminal open while mixing)
cd C:\Users\arnau\Documents\deckstream
python serato_event_agent.py

You should see:

INFO  Serato DB found — no active session yet (open Serato + load a track)
INFO  Kafka producer connected to localhost:19092
INFO  Watching : C:\Users\arnau\AppData\Local\Serato\Library

3. Open Serato DJ and load a track on deck 1

The agent will print:

INFO  Track changed → Your Track Title
INFO  → Produced to serato.track.events: /music/GENRE/track.mp3

Recommendations appear at http://localhost:8050 within ~1 second.

---

Services

URL	Service	Purpose
http://localhost:8050	Dashboard	DJ view + library management
http://localhost:8080	Spark UI	Spark cluster status
http://localhost:8090	Kafka UI	Topics, consumer lag, message payloads, latency

Kafka UI — latency monitoring

Every recommendation message on serato.recommendations contains:

• agent_ts — when the agent detected the track change
• spark_ts — when Spark finished scoring

spark_ts − agent_ts = pipeline latency (typically 0.5–1.2 s).

---

Docker commands

# Normal start
docker compose up -d

# Rebuild after Dockerfile or dependency changes
docker compose build
docker compose up -d

# Force-recreate a specific service (picks up env var changes)
docker compose up -d --force-recreate dashboard
docker compose up -d --force-recreate stream-processor

# Restart streaming pipeline only
docker compose restart stream-processor

# Stop everything
docker compose down

# Follow logs
docker logs -f deckstream-dashboard
docker logs -f stream-processor
docker logs -f redpanda

# Check all container statuses
docker compose ps

---

Remote access (Tailscale)

Access the dashboard from your phone or any device without port forwarding.

Setup

# Install Tailscale on Windows
winget install tailscale.tailscale

Then install Tailscale on your phone (App Store / Play Store) and log in with the same account.

Usage

1. Find your PC's Tailscale IP at login.tailscale.com/admin/machines (format: 100.x.x.x)
2. On your phone: http://100.x.x.x:8050

Requirements:

• docker compose up -d must be running on the Windows host
• Tailscale must show "Connected" in the Windows system tray
• The Docker port binding must be 0.0.0.0:8050 (not 127.0.0.1) — check docker-compose.yml

---

Ingesting your music library

Three phases — always run in order from the dashboard container:

# Phase 1 — scan audio files → extract metadata → Delta Lake
docker exec deckstream-dashboard python -m src.ingestion.cli ingest /music/HOUSE

# Phase 2 — generate acoustic embeddings (requires embedder container + GPU)
docker exec deckstream-dashboard python -m src.ingestion.cli embed /music/HOUSE

# Phase 3 — merge embeddings into Delta Lake
docker exec deckstream-dashboard python -m src.ingestion.cli merge

# Full pipeline in one command
docker exec deckstream-dashboard python -m src.ingestion.cli all /music/HOUSE

Or use the Library page at http://localhost:8050/library — paste a path, click Ingest.

---

Project structure

deckstream/
├── serato_event_agent.py          # Windows host agent (ReadDirectoryChangesW → Kafka)
├── src/
│   ├── dashboard/
│   │   ├── app.py                 # App factory, clientside SSE callback
│   │   ├── sse.py                 # SeratoSSEBroadcaster (Kafka consumer + fallback)
│   │   ├── scoring.py             # Hybrid additive scoring
│   │   ├── recommendation_engine.py  # Camelot Wheel 7-transition engine
│   │   ├── delta_utils.py         # PySpark Delta Lake queries + 3-layer cache
│   │   ├── serato_sqlite.py       # WAL-safe Serato SQLite reader
│   │   ├── layout.py              # All page layouts
│   │   ├── art.py                 # Album art cache
│   │   ├── colors.py              # Design tokens
│   │   ├── assets/
│   │   │   ├── style.css          # Glassmorphic design system
│   │   │   └── sse_client.js      # Browser EventSource client
│   │   └── callbacks/
│   │       ├── current_track.py   # Now Playing (fast path: Kafka / slow path: SQLite)
│   │       ├── recommendations.py # Recommendation rendering (fast path: Kafka payload)
│   │       ├── history.py         # Play history
│   │       ├── library.py         # Ingest / remove / embed jobs
│   │       └── routing.py         # Page routing
│   ├── streaming/
│   │   └── spark_recommender.py   # Spark Structured Streaming job
│   └── ingestion/
│       ├── cli.py                 # Entry point: ingest / embed / merge / all
│       ├── ingest.py              # Bulk PySpark ingestion (IngestionRunner)
│       ├── embedder.py            # Essentia discogs-effnet acoustic embeddings
│       ├── metadata_extractor.py  # mutagen tag extraction (MP3, FLAC, WAV)
│       ├── hierarchy_parser.py    # Path → party_type / genre / sub_genre / vibe
│       └── schema.py              # Spark schemas + Pandera validation
├── docs/
│   ├── schemas/
│   │   ├── live-workflow.md       # Mermaid: end-to-end live pipeline
│   │   └── ingestion-workflow.md  # Mermaid: three-phase ingestion
│   ├── EN/                        # English documentation
│   └── FR/                        # French documentation
├── data/                          # Delta Lake warehouse (git-ignored)
├── config/                        # config.yaml (scoring weights, paths)
├── Dockerfile                     # Spark + dashboard + ingestion image
├── Dockerfile.embedder            # Essentia GPU image
└── docker-compose.yml             # Full stack definition

---

Tech stack

Layer	Technology	Why
Event broker	Redpanda (Kafka-compatible)	Single container, no Zookeeper, ~512 MB RAM, <1 ms latency
Stream processor	Spark Structured Streaming	500 ms micro-batch, Delta Lake integration, Camelot scoring
Storage	Delta Lake (PySpark)	ACID transactions, schema enforcement, time travel
Backend	Python 3.11, PySpark 3.5	—
Frontend	Dash 4.1, Plotly	Reactive Python UI, SSE clientside callbacks
Live tracking	Serato SQLite (master.sqlite)	Serato DJ 4+ drops binary .session files, uses SQLite
Acoustic similarity	Essentia discogs-effnet	200-dim embedding, GPU inference
Containerisation	Docker, Docker Compose	—
Remote access	Tailscale (WireGuard)	Phone access without port forwarding

---

Documentation

File	Content
CLAUDE.md	Full development guide — architecture, pitfalls, design decisions
docs/schemas/live-workflow.md	Mermaid diagram — live pipeline
docs/schemas/ingestion-workflow.md	Mermaid diagram — ingestion
docs/EN/ARCHITECTURE.md	System architecture + latency budget
docs/EN/STREAMING.md	Kafka/Redpanda/Spark deep dive (Netflix, Spotify, Uber, LinkedIn comparisons)
docs/EN/NOW_PLAYING.md	Live detection implementation details
docs/EN/TECH_STACK.md	Every tool and why it's there
docs/EN/SCORING.md	Recommendation scoring algorithm
docs/EN/CAMELOT_WHEEL.md	Camelot Wheel key compatibility reference
docs/EN/INGESTION.md	Ingestion pipeline guide
docs/EN/TAILSCALE.md	Remote access setup