Music Similarity Based on Multivariate Correlation

Music similarity using statistical methods (PCA + Earth Mover's Distance / Mutual Information) as an alternative to neural embedding models. Evaluated on the MagnaTagATune odd-one-out task.

Full details in the project report.

Setup

The project requires two data artifacts before the web app can run. Both are too large to commit.

1. MTAT dataset in mtat/

Download the MagnaTagATune audio files (~3 GB, 26k clips of 30 seconds):

cd src
python mtat_download.py        # downloads and extracts into ../mtat/

This creates:

mtat/
├── 0/ … 9/                    # MP3 audio clips organized in subdirectories
├── annotations_final.csv      # clip_id, mp3_path, 190 binary tag columns
└── comparisons_final.csv      # triplet clip IDs with odd-one-out votes

The audio files are needed by the backend to stream reference songs for playback in the UI.

2. Reference dataset in src/backend/reference_dataset/

Pre-computed models and features used by the backend at startup for similarity queries. Build it from the MTAT dataset:

cd src/backend
uv run python scripts/build_reference_dataset.py \
  --audio-dir ../../mtat \
  --scope comparisons \
  --comparisons-csv ../../mtat/comparisons_final.csv \
  --clip-info-csv ../../mtat/annotations_final.csv \
  --output-dir reference_dataset

This produces:

src/backend/reference_dataset/
├── manifest.json       # song_id → filename, mp3_path, n_frames
├── scaler.pkl          # fitted StandardScaler
├── pca.pkl             # fitted IncrementalPCA (10 components)
├── kmeans.pkl          # fitted global K-Means (24 clusters)
└── features/           # PCA-transformed features, one .npy per song

3. Configure the audio base path

The backend needs to know where the MTAT audio files live on disk to stream them. Set the environment variable before starting the server:

export APP_REFERENCE_AUDIO_BASE_PATH=/absolute/path/to/mtat

This resolves the relative mp3_path values from manifest.json to actual files on disk. If not set, it defaults to a hardcoded developer path that won't work on your machine.

4. Optional: preprocessing model artifacts (grid search / evaluation)

A handful of large model/cache files are not committed (GitHub blocks files >100MB) and are only needed for the EMD grid-search and cosine-similarity evaluation notebooks, not for running the web app.

Global K-means models — src/preprocessing/kmeans_25k_{2,3,5}d_{24,32}.pkl (6 files, used by src/evaluation/emd/):

# Requires step 2 above to be done first (needs src/backend/reference_dataset/features/)
cd src/preprocessing
jupyter execute fit_kmeans.ipynb

Fits global K-means over all ~25,860 songs' full 30-component features, once per (n_components, n_clusters) combination in {2,3,5} x {24,32}. Each configuration took ~9 minutes in the original run over 52.6M frames, so budget an hour or more for all 6.

PCA-transformed triplet/song caches — src/outputs/transformed_triplet_10components.pkl, cache/transformed_songs_10components.pkl, cache/transformed_triplet_10components.pkl:

Repository structure

docs/
├── Report/                     # LaTeX project report
│   └── MultiCorr_Report.tex
├── papers/                     # Reference papers
└── MultiCorr-Presentation.pptx

src/
├── backend/                    # FastAPI backend (see backend/README.md)
├── frontend/                   # Next.js frontend (see frontend/README.md)
├── evaluation/                 # Evaluation notebooks + results
│   ├── emd/                    #   EMD grid search, evaluation, timing
│   ├── mi/                     #   MI evaluation, timing benchmarks
│   └── neural/                 #   MERT, CLMR, CLAP baselines
├── preprocessing/              #   PCA/K-Means model fitting notebooks
├── outputs/                    #   Report figures, intermediate artifacts
├── mtat_download.py            # MTAT dataset downloader
└── Makefile                    # Dev commands (setup, dev, clean)

mtat/                           # MTAT audio + annotations (not committed)

Web App Quick start

Prerequisites: uv, Node.js (v18+).

cd src
make setup          # install backend + frontend dependencies
make dev            # start both servers (backend :8000, frontend :3000)

Or run them separately:

make dev-backend    # backend only  → http://localhost:8000
make dev-frontend   # frontend only → http://localhost:3000

The backend requires a pre-built reference dataset at src/backend/reference_dataset/. See src/backend/README.md for build instructions.