beatmap-visualizer

A standalone previewer for what a light show driven by a beatmap (or, later, a live salience model) would actually look like. Built to work directly on real osu! beatmaps first, with zero dependency on any audio-to-salience model or training pipeline -- ground truth from [HitObjects] is unambiguous, so this is where the chart/light-preview UX gets validated before anything gets wired up to live audio.

Why this is a separate project

It's tempting to build "stream audio -> features" and "preview the light show" as one thing, but that gets the order backwards: you can't tell if a light-mapping decision (decay curve, gamma, color) is right if you're simultaneously debugging whether the model's signal is right. This project was built beatmap-first so the preview/UX work could be validated against unambiguous ground truth before anything got wired up to live audio. Model mode and live capture mode (see "Model mode" below) both exist now, but both still emit onset/sustain/energy in that exact same shape, so they plug into the same chart/light-preview code the ground-truth path validated.

Run it

pip install -r requirements.txt
python server.py

Requires ffmpeg on PATH (used to decode whatever audio format is inside the .osz). Drop a few .osz files into maps/ first -- the server scans that folder on startup.

Alternatively, use the folder picker at the top of the page to point the server directly at a real osu! Songs folder (e.g. C:\Program Files\osu\Songs) -- no need to export/copy .osz files at all. discover_songs() understands both layouts: flat .osz archives, and real osu! Songs folders (one subfolder per beatmapset, .osu + audio sitting uncompressed inside). The chosen folder is remembered across restarts in a local, gitignored .config.json; --songs-dir PATH overrides it from the command line.

"Browse folder…" / "Browse .osz file…" open a native OS picker dialog (via tkinter, so no manual path-typing is required) instead of the text box -- picking a single .osz file just scans its parent folder, same as pointing the folder picker at it directly.

Scanning a folder only reads each .osu's metadata (audio filename, version tag, hit-object count) -- it does not resolve slider end times, which is the expensive part of a full parse and is unnecessary for just listing songs. The one difficulty you actually select gets a full parse (slider durations, sustain envelope, etc.) at that point. This keeps scanning a multi-thousand-difficulty real osu! Songs folder to single-digit seconds instead of a minute-plus.

The picker lists one row per song (beatmapset), not one row per difficulty -- picking a song reveals a difficulty slider underneath, sorted from fewest hit objects to most (a free proxy for "easiest first" since no star rating is computed here) and defaulting to that easiest difficulty.

What it shows

1. Data -- onset + sustain curves built straight from the chosen difficulty's real [HitObjects] (signals.build_target_and_mask). Ground truth renders as sharp, near-0/1 impulses. A future model-driven version would feed continuous 0-1 certainty into this exact same chart.
2. In-between signals -- starts with Energy, a per-frame RMS loudness curve computed directly from the decoded audio (signals.energy), no beatmap or model involved. More signals (e.g. spectral brightness) can be added here the same way.
3. Light show result -- the onset/sustain curves combined into a single light-output curve (flash + exponential decay, held open by sustain). The decay half-life set by the slider (default 200ms) is itself modulated by the current Energy value: louder moments fade back to 0 up to 2x faster, quiet moments hold the glow up to 2x longer -- so the light's responsiveness tracks how much is going on in the audio, not just the beatmap. The "Sustain blend" slider (default 1.0) controls how much a held sustain overrides that decay entirely. A peak-follower on Energy (same flash+decay shape as the light itself -- instant attack, exponential release with the half-life set by the "Energy ceiling decay" slider, default 4s) then caps how bright the light is allowed to get -- a strong onset during a quiet passage stays dim, a loud passage (a drop, a peak moment) opens the ceiling, and it stays open for a while after as the follower decays back down; the flash/decay/sustain dynamics above are unaffected, only the final brightness ceiling is. This ceiling is plotted as a dashed line directly on the light chart (gamma-corrected to the same space as the light curve, so it renders as a true visual cap the light curve never crosses). The result is then gamma-corrected (value ** gamma, slider range 0-5, default 3.4) as the final step -- higher gamma crushes low values harder, tune it to match how the actual light/screen responds. The swatch color hue is user-adjustable (it's simulating a colored stage light). Double-click the swatch, or use the "Fullscreen light" button, to blow it up to fill the whole screen.

All three panels sit side by side so they're easy to compare against each other at a glance (they wrap onto their own row on narrow windows).

Dark theme always, everywhere, except the light swatch's own hue -- this tool exists to preview how a light looks in a dim room.

Model mode

A "Visualize using:" selector above the song picker switches panels 1-2 between two sources, without changing anything else (charts, light combiner, Art-Net, fullscreen all behave identically either way):

• Real beatmap (default) -- unchanged ground truth from [HitObjects], as described above.
• Model -- model.py's estimate_curves(), computed from the decoded audio alone; the .osu file is never read in this mode. Runs the trained SalienceTCN checkpoint from osu2light-main (salience_tcn.py for the architecture, mel_features.py for the causal log-mel frontend -- vendored copies so this project doesn't depend on the osu2light package) -- both onset and sustain come out of the model directly, no placeholder math. The checkpoint is conditioned on a "difficulty target" d in [0, 1], exposed as a slider in the UI (&d= on /api/curves, only meaningful in Model mode). The checkpoint file itself lives outside this repo (it's a large binary from the training project); its path defaults to model.DEFAULT_CHECKPOINT_PATH and is overridable via the BEATMAP_VIS_MODEL_CHECKPOINT env var. Normalization constants (feature_mean/feature_std/d_low/d_high, fixed at training time) are checked into model_norm.json.

Both sources are served from the same /api/curves?song=ID&source=beatmap|model endpoint, shaped like:

{
  "hop_ms": 10,
  "duration_ms": 123456,
  "onset": [0.0, 0.02, ...],
  "sustain": [0.0, 0.0, ...],
  "energy": [0.1, 0.15, ...],
  "source": "beatmap"
}

Same frame grid (hop_ms), same [0, 1] range per array, same length (duration_ms / hop_ms + 1) regardless of source, so static/app.js's chart/combiner code needs no per-source branching.

Live mode: tab/screen-capture audio instead of a song file

A third "Visualize using:" option, Live, skips the mp3 entirely -- instead of picking a song, click "Start capture…" and pick a Chromium tab (with "Share tab audio" enabled), window, or screen in the browser's own share picker. This is getDisplayMedia({ video: true, audio: true }) (the video track is stopped immediately, audio only) -- not microphone input, which was tried elsewhere and is unreliable.

Unlike beatmap/model mode, live mode never touches server.py's /api/curves endpoint -- everything runs client-side in static/app.js: a Web Audio AnalyserNode reads the captured track, and a lightweight spectral-flux onset detector runs per ~10ms hop directly in the browser (the trained checkpoint Model mode uses needs torch and a fully-decoded file, not viable per-frame in the browser). Onset/energy are normalized against a decaying peak-follower instead of a whole-file percentile, since a live stream has no "whole file" to look ahead into. Output feeds the same state.onset / state.sustain (always 0, same as model mode) / state.energy arrays that file-backed sources populate, just as fixed-length rolling ring buffers (last 8 seconds) instead of one-shot arrays -- so recomputeLight(), the three charts, the brightness swatch, and Art-Net output all run completely unmodified in live mode.

Layout

• osu_parser.py -- .osu file parsing (HitObjects, breaks, timing points for slider-duration math only).
• signals.py -- onset/sustain/ignore-mask construction from HitObjects, plus energy() from raw audio.
• model.py -- audio-only onset/sustain estimate from the trained SalienceTCN checkpoint, used by "Model" source mode; see "Model mode" above. salience_tcn.py / mel_features.py hold the vendored architecture / feature frontend it depends on; model_norm.json holds the training-time normalization constants.
• audio.py -- ffmpeg-based audio decode.
• server.py -- stdlib http.server, scans maps/*.osz, serves the API + static frontend.
• static/ -- index.html / app.css / app.js, the chart + light combiner, no build step, no framework.
• maps/ -- drop .osz files here (gitignored -- these are copyrighted beatmap archives, don't commit them).

Genres: validated primarily against Hard Techno, Hardstyle, and Vocaloid maps, but nothing in the signal math or UI is genre-specific -- put maps from other genres in maps/ too and it should work the same.

Out of scope

• Training or evaluating the model, or reproducing osu2light-main's dataset pipeline -- this project only runs inference against an already-trained checkpoint (see "Model mode" above).
• DMX/ArtNet/actual lighting hardware control -- this is a browser preview only.