viz: music-synced timeline playback + onset-timings extraction#65
Open
kumavis wants to merge 7 commits into
Open
viz: music-synced timeline playback + onset-timings extraction#65kumavis wants to merge 7 commits into
kumavis wants to merge 7 commits into
Conversation
Add an alternate play mechanism to the propagator-network visualizer that advances the execution timeline in time with a song: load an audio file and a timings file (hit onsets in seconds), play the audio, and each onset advances one step. Runs alongside the existing timer ▶ control (independent; starting audio pauses the timer). - index.html: parseTimings + stepForTime pure helpers (node-verified in check.js); App-level audio/timings state + a requestAnimationFrame loop that maps audio.currentTime → step index via the onsets; a MusicSync sidebar panel (audio + timings file pickers, an <audio> element, status readout). - Legend now sticks to the bottom of the sidebar (sticky bottom + flex column). - extract-song-timings.py: URL/file → yt-dlp → ffmpeg → spectral-flux onset detection → timings JSON. Tuned for sharp percussive (xylophone) hits; numpy only, no librosa. - examples/: a small copyright-free synthetic xylophone demo (mp3 + timings) plus make-demo-song.py to regenerate it; the detector recovers all 13 onsets, so it doubles as an end-to-end pipeline check. - README.md documenting playback, timings format, extraction, and the demo. Timings format accepts a bare JSON array, an object keyed by hits/timings/beats/onsets/times, or whitespace-separated plain text.
The actual song audio was provided, so extract real timings from it. - extract-song-timings.py: add --hifreq, weighting spectral flux toward high frequencies (>~1.5 kHz) to isolate bright mallet/xylophone/glockenspiel hits from bass/kick/vocals. - examples/praise-the-lamb.timings.json: 1549 --hifreq onsets from Cult of the Lamb — Praise the Lamb (youtu.be/PoH5hC5PzSQ). Audio not committed (copyright); pair with your own copy in the music panel. - README: document --hifreq and the real timings file.
…trings) Replace the crude --hifreq flag (>1.5 kHz, which would lock onto the arpeggiated strings) with frequency-band selection. Spectrum analysis of the opening hit (xylophone note + cymbal crash) shows the xylophone fundamental at ~145 Hz, so the melodic line sits low. - extract-song-timings.py: --band-low / --band-high (Hz) restrict the spectral flux to one instrument's band; isolates the low xylophone from kick/bass below and strings above. - examples/praise-the-lamb.timings.json: regenerated in the low band (--band-low 120 --band-high 520). Band/threshold still being auditioned against the audio — may be refined. - README: document band selection.
Confirmed by audio audition: band 110–300 Hz (threshold 1.5, min-gap 0.11) tracks the low xylophone. Regenerate examples/praise-the-lamb.timings.json (1690 onsets) and set the README example to the confirmed settings.
The 110–300 Hz band still caught quiet bass/percussion bleed (~4.7/s); the
real low xylophone is ~1-2 notes/s ("bong bing bong bing"). Add a --min-strength
loudness floor (fraction of the 99.5th-pct flux) that keeps only the loud mallet
hits while preserving the close bong+bing pairs (unlike a larger min-gap, which
merges them).
- extract-song-timings.py: add --min-strength gate.
- examples/praise-the-lamb.timings.json: regenerate with
--band-low 110 --band-high 300 --min-strength 0.5 --min-gap 0.12 → 634 onsets
(~1.8/s; intro ~1.7/s, clean bong-bing-bong-bing).
- Untrack a stray __pycache__/*.pyc and gitignore Python bytecode.
- README: document --min-strength and the updated settings.
…cking The low xylophone is metronomic but quieter than the kick/snare on the loud beats, so onset-picking grabbed the loud off-beat transients and missed the quiet xylophone notes (e.g. detected 1.44/1.67/2.32 where the real notes are 1.90/2.80). Waveform ground truth showed a steady ~0.89s pulse. Add --grid mode: estimate tempo from the band onset-envelope autocorrelation, anchor the phase to the instrument's first hit (--phase; blind phase-finding locks onto the louder beat), and emit phase + k·period. Joint period+phase refinement keeps a constant-tempo track aligned with no drift over its length (verified to ±15ms at the 5-minute mark). - extract-song-timings.py: band_onset_envelope + beat_grid; --grid/--bpm/--phase /--tempo-min/--tempo-max; record mode/bpm/period/phase in the JSON. - examples/praise-the-lamb.timings.json: regenerate in --grid mode → 407 beats @ 67.5 BPM (~1.1/s), matching the waveform (0.08, 0.97, 1.86, 2.75, 3.64…). - README: document beat-grid mode and the tradeoff (pulses through rests).
Alternate playback take: hold the slow xylophone pulse through the intro, then rush at the drop. Add --segments 'startSec:bpm,...' for a piecewise-constant beat grid; a tempo switch lands a beat exactly on the boundary. - extract-song-timings.py: tempo_map() + --segments; record segments in the JSON. - examples/praise-the-lamb-tempomap.timings.json: '0:67.5,28.5:675' → slow 67.5 BPM intro, 10x (675 BPM) burst from 28.5s (3773 beats). - README: document --segments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Targets the PR #64 branch (
claude/charming-archimedes-98yb48).What this adds
An alternate play mechanism for the propagator-network visualizer that advances the execution timeline in time with a song — load an audio file + a timings file (hit onsets in seconds, e.g. a song's xylophone hits), play the audio, and each onset advances one step, so the on-network reduction plays back synced to the music.
It sits next to the existing timer-based
▶control and is fully independent (starting the audio pauses the timer).tools/viz/index.htmlparseTimings+stepForTime— pure, node-verified helpers in the@PUREcore.requestAnimationFrameloop mappingaudio.currentTime→ step index via the onset list (binary search).<audio>element, and a live status readout.tools/viz/extract-song-timings.pyURL/file →
yt-dlp→ffmpeg→ spectral-flux onset detection → timings JSON. Tuned for sharp percussive (xylophone) hits, with--threshold/--min-gapknobs.numpyonly — no librosa.tools/viz/examples/A small, copyright-free synthetic xylophone demo (
demo-xylophone.mp3+demo-xylophone.timings.json) plusmake-demo-song.pyto regenerate it. The detector recovers all 13 onsets within ~30 ms, so the demo doubles as an end-to-end pipeline check. Drop a vizTrace, then load these two files in the music panel and press play.Timings file format
Accepts a bare JSON array
[0.30, 0.62, …], an object keyed byhits/timings/beats/onsets/times, or whitespace/comma/newline-separated plain text.On the YouTube extraction
The request was to pull audio from
https://youtu.be/PoH5hC5PzSQand extract xylophone-hit timings. I built and validated the fullyt-dlp → ffmpeg → onset-detectionpipeline, but the actual download is blocked from this environment: YouTube serves only PO-token/DRM-gated formats to the datacenter IP and falls back to "confirm you're not a bot." The pipeline runs cleanly on a machine with a logged-in browser via--cookies-from-browser:So the committed demo timings are the synthetic phrase (clearly labeled), and the real song's timings are one command away on a browser-equipped machine.
Verification
node tools/viz/check.js tools/viz/index.html→ ALL CHECKS PASS (incl. newparseTimings/stepForTimeunit tests).<script type="text/babel">block transforms cleanly under@babel/preset-react(no JSX/syntax errors).https://claude.ai/code/session_01HcdtkcuDMUFokytmRhkQcK
Generated by Claude Code