avatar-sync

Real-time face + hand tracking in your browser. One HTML file. No server, no API keys, no install.

Open the page, allow the camera, and you get:

• 478 face landmarks drawn live on your face
• Emotion readout — HAPPY / SURPRISED / SAD / NEUTRAL
• Head pose — yaw, pitch, roll in degrees
• Expression meters — blink, brow, smile, mouth-open, pucker (0–100%)
• Both hands tracked — 21-point skeleton, fingertip labels, OPEN/CLOSED detection
• ⏺ Record button — captures a 10s clip of the annotated canvas and downloads it as .webm

Everything runs on-device with MediaPipe Tasks Vision — nothing leaves your machine.

Quick start

git clone https://github.com/rohitguta2432/avatar-sync.git
cd avatar-sync
python3 -m http.server 8772

Open http://localhost:8772, click Allow on the camera prompt. Done.

Any static file server works — the only requirement is serving over http://localhost or https:// (browsers block camera access on file://).

How it works

The heavy lifting is two tiny pre-trained models that download into the browser on first load:

Model	Gives you
FaceLandmarker	478 face points + 52 expression scores ("blendshapes") + head transform
HandLandmarker	21 points per hand + left/right handedness

The app is just a requestAnimationFrame loop: draw the camera frame, run both models on it, draw dots/lines/labels on a canvas. The "SMILE 35%" you see is literally the model's mouthSmileLeft score printed on screen; the emotion label is a one-line threshold check. Hand OPEN/CLOSED is plain geometry — a hand is open when 3+ fingertips are farther from the wrist than their middle joints.

Use it as a base for

• VTuber-style avatar puppeteering (pipe the blendshapes into a Live2D/VRM rig with Kalidokit)
• Gesture-controlled UIs
• Attention / drowsiness detection experiments
• Filming tech demos 😉

License

MIT © Rohit Raj