AI web application that turns audio files (MP3) into sheet music by separating songs into individual instrument tracks and generating interactive notation.
Splitsound is a modern full-stack web application designed for musicians, arrangers, and music enthusiasts. It allows users to upload audio tracks, automatically separate them into individual stems (Vocals, Bass, Drums, Guitar/Piano) using AI, and transcribe those stems into interactive MusicXML sheet music directly in the browser.
- AI Audio Separation: Splits stereo audio tracks into discrete stems (Vocals, Bass, Drums, Guitar/Piano) powered by Meta's Demucs model.
- Automated Transcription: Converts audio stems into MIDI and MusicXML sheet music using Spotify's Basic Pitch neural network and Music21.
- Interactive Notation View: Renders dynamic, clean sheet music directly in the frontend using Open Sheet Music Display (OSMD).
- Real-Time Job Tracking: Live progress updates during backend AI processing tasks.
- User Library & Storage Management: Track upload history, storage usage, and manage processed jobs seamlessly.
- Modern Dark UI: Responsive and sleek user interface built with React, TypeScript, and Tailwind CSS.
Splitsound operates on a decoupled micro-architecture where heavy machine learning inference is offloaded to asynchronous background processing workers:
[ User Audio Upload ]
│
▼
[ React Frontend ]
│ (REST API)
▼
[ FastAPI Backend ] ──► [ Async Worker Task Queue ]
│
┌────────────────────────────┴────────────────────────────┐
▼ ▼
[ 1. Demucs AI ] [ 2. Basic Pitch AI ]
Isolates audio stems Audio-to-MIDI pitch
(Vocals, Bass, Drums, etc.) detection per stem
│ │
└────────────────────────────┬────────────────────────────┘
▼
[ 3. Music21 Engine ]
Converts MIDI into
standard MusicXML format
│
▼
[ Frontend Rendering ]
OSMD renders score
interactively in UI
- Framework: React 18 (TypeScript, Vite)
- Styling: Tailwind CSS
- Notation Engine: Open Sheet Music Display (OSMD)
- Routing & State: React Router, Context API
- API Framework: Python / FastAPI
- Audio Separation Engine: Meta Demucs v4
- Pitch Detection Engine: Spotify Basic Pitch
- Music Theory & XML Processing: Music21
- Audio Processing Utilities: Librosa, SoundFile, FFmpeg
- Node.js (v18+ recommended)
- Python (v3.10+ recommended)
- FFmpeg installed on your system PATH
-
Clone the repository:
git clone https://github.com/Smelda07/Splitsound.git cd Splitsound -
Run the startup script:
chmod +x start.sh # Ensure execution permissions (if needed) ./start.sh
The start.sh script automatically handles dependencies, initializes local environments, and launches the Frontend, Backend, and AI Worker services simultaneously.
- Open application:
Navigate to
http://localhost:5173in your browser.
Splitsound/
├── backend/ # FastAPI REST backend and API routes
├── frontend/ # React + TypeScript frontend application
├── worker/ # Asynchronous background scripts for Demucs & Basic Pitch
├── docs/ # System architecture and documentation resources
└── start.sh # One-click startup script for all services
- Advanced Pitch Quantization: Integrate smart rhythm quantization via Music21 to eliminate micro-rests.
- Interactive Audio Player: Synchronize audio playback with real-time sheet music cursor scrolling.
- Multi-Format Export: Allow users to export transcribed tracks as
.midi, PDF, or raw.musicxml.