Expression Reader

Teach a classifier your own face, in about ten seconds, in your browser.

▶ Open the live demo — a model has already trained before you arrive, no camera needed

---

The features are already good, so the model can be small

MediaPipe returns 52 ARKit-style blendshape scores per frame — how open the jaw is, how raised the left brow is, how far each mouth corner has pulled. That is a well-engineered 52-dimensional feature vector arriving at 30 fps, and it turns expression recognition from a computer vision problem into a plain supervised classification problem.

Which means the model doesn't need to be a CNN. It's a multinomial logistic regression trained with Adam, written from scratch — no TensorFlow.js, no autograd. The gradient of cross-entropy through a softmax is (p − y), which is one line, and the whole thing trains in 33 ms on 225 samples.

Four details that decide whether a hand-written trainer works

Features are standardised first. Blendshapes are all nominally 0–1, but their real spread differs by an order of magnitude — jawOpen swings across most of its range while noseSneerLeft barely moves. Unscaled, one step size cannot suit both and training oscillates.

The softmax is max-shifted. Without it, exp() of a confident logit overflows to Infinity and every probability comes back NaN — a bug that hides well, because argmax over NaNs still returns something.

Early stopping keeps the best-validating weights, not the final ones. Otherwise the accuracy you report is whatever the last epoch happened to land on.

The bias is not regularised. Penalising it just biases the model away from the true class priors.

It refuses to learn from noise

Trained on 240 random vectors with random labels:

[ok] does not learn from noise — val acc 31.7% (chance is 33%)
[ok] but still overfits the training split — train 42.8% vs val 31.7%

It memorises the training data and generalises not at all, which is exactly what it should do — and is the whole reason the held-out split exists. 26 checks in CI cover softmax overflow, the scaler's divide-by-zero guard, early stopping returning the best weights, confusion matrix arithmetic, and determinism.

There is a rule baseline, and it is labelled as rules

Alongside the trained model, the page runs a hand-written mapping — smile blendshapes mean happy, brow down means angry. It is not learning and it says so wherever it appears. Its job is to be the thing the trained model has to beat, because without a baseline "87% accurate" is a number with no scale attached.

It works without a camera

On load, a classifier trains on a synthetic blendshape dataset — the faces are invented, the training is the same code that runs on yours. So the machine learning is visible before anyone is asked for camera permission.

The synthetic problem is deliberately not easy. My first version produced 98% held-out accuracy, which makes a pretty demo and an uninstructive one: a confusion matrix with nothing off the diagonal teaches the reader nothing about where a model fails. Tuning the noise and adding cross-talk between expressions — real faces are rarely pure — lands it at:

Training accuracy	100%
Held-out accuracy	87.5%
Training time	33 ms, 70 epochs

The gap between those two numbers is the point.

Run it

git clone https://github.com/Devapriyan-S/expression-reader.git
cd expression-reader

npm test                            # 26 checks, no browser needed
python -m http.server 8000 -d web   # open http://localhost:8000

The classifier is a dependency-free ES module and works on any feature matrix:

import { SoftmaxRegression } from "./web/js/classifier.js";

const model = new SoftmaxRegression(nFeatures, nClasses);
const report = model.fit(X, y);            // { trainAcc, valAcc, epochs, loss }

model.predict(row);                        // { label, confidence, probabilities }
model.confusion(X, y);                     // k x k, rows = truth
model.explain(row, classIndex);            // weight x scaled input, per feature

Limits

• Five expressions, one face. Neutral, happy, sad, surprised, angry, and numFaces: 1.
• A linear model on blendshapes. It cannot represent an expression that depends on a combination of shapes rather than their weighted sum. A small MLP would, at the cost of needing more than forty samples per class.
• Blendshapes carry the model's biases. MediaPipe's estimates are less reliable with glasses, heavy beards, extreme head pose or low light, and nothing here corrects for that. An expression the model reads badly may be a feature problem, not a classifier problem.
• Samples live in memory only. Reload the page and your recordings are gone; nothing is persisted and nothing is uploaded.

---

MIT licensed. Built by Devapriyan Sampath — portfolio · LinkedIn · devapriyan1723@gmail.com