Dextera 🎹

Music that moves with you.

Dextera makes piano practice more accessible for people with Parkinson's disease. Around 7–10 million people worldwide live with Parkinson's, and up to 90% experience bradykinesia (slowness of movement) and tremors. Dextera combines a simplified on-screen keyboard with camera-based Hand Mode, so you can play notes using hand gestures instead of touch.

Dextera is a  2026 Apple Swift Student Challenge Winner and is featured on the WWDC Scholars page!!

Key Features 🎼

• Guided practice — Step-by-step note highlighting for familiar songs (Twinkle Twinkle, Mary Had a Little Lamb, Ode to Joy, Scales). Sessions are saved automatically.
• Free play — Play freely on the keyboard with touch, or switch to Hand Mode and play using hand gestures through the camera.
• Hand Mode — Uses the Vision framework for real-time hand pose detection. Map fingers to notes and play without pressing the screen.
• Past practice — Track sessions, streaks, and a song leaderboard of your favourites.
• Learn — About Parkinson's, bradykinesia, why music matters, and what the research shows.
• Accessibility Features — Adjustable brightness, text size, sound and haptic toggles.

Technologies Used 🔧

• Swift & SwiftUI
• AVFoundation & AVAudioEngine — Camera capture, audio synthesis and piano sounds
• Vision — Hand pose detection
• Swift Charts - Visualizes progress + song leaderboard
• Combine - Processes values

Why Dextera? 🩵

Dextera was built for my grandfather, who loved music but could no longer play physically due to bradykinesia and tremors from Parkinson's disease. Traditional piano apps require precise key presses, which becomes difficult when movement is slow or shaky.

Research from the Parkinson’s Foundation, UCL, and other institutions suggests that music therapy can help reduce bradykinesia and support motor function, gait, speech, and mood. Dextera is designed to adapt to different stages of motor control. Users in earlier stages can play by tapping the on-screen keys with their fingers. For those who find precise tapping challenging, Hand Mode offers an alternative: point a finger toward the camera to trigger a note, without needing to press the target on the screen.

Dextera was designed to give people with limited dexterity a way to play their favourite songs again.