A touch-free, predictive text-entry system that translates mid-air hand gestures into text — no physical contact required. Built for accessibility, it uses an 8-directional gesture model, a Hidden Markov Model with Viterbi decoding, and a custom-optimized keyboard layout to handle real-world hand noise, tremors, and spatial variance.
This project started as a tool for people who couldn't easily access a physical keyboard — hands that are dirty, occupied, or unsteady. Through research in the HCI field, the design pivoted from letter-by-letter air-handwriting (slow, error-prone, physically exhausting) toward a directional input model: 8 gestures map to groups of letters, and a machine learning backend decodes the intended word. The result is a system that is fast enough for daily use and accessible enough for users with motor impairments such as Parkinson's disease.
MediaPipe's Hand Landmark model detects 21 keypoints on the hand in real time. The index finger is isolated and tracked across frames.
Gestures are classified into 11 inputs:
- 8 directional swipes (up, down, left, right, and four diagonals) — each maps to a group of 3–4 letters
- Clockwise arc → Space
- Counter-clockwise arc → Backspace
- Z-axis push (depth from wrist) → Enter
Intentional movements are separated from drift and micro-tremors using two criteria derived from Rubine's gesture recognition framework:
- Directness ratio: if the ratio of straight-line distance to total path length falls below 0.85, the movement is classified as drift
- Macro-velocity threshold: a keystroke is only registered when a clear directional movement crosses a minimum speed and distance threshold, filtering out hand tremors
This creates an effectively infinite target area — removing the strict accuracy penalties of small virtual buttons — and makes the system usable for people with tremors or limited motor control.
The 26 letters are grouped into 8 keys. A custom scoring algorithm evaluates layouts on two criteria:
- Substitution traps (high severity): multiple common words sharing the same gesture code, making decoding ambiguous
- Double-click traps (low severity): consecutive gestures on the same key, which slows input but doesn't cause decoding errors
Layouts are scored using letter and bigram frequencies from Google Corpus Data, then iteratively improved with a hill-climbing algorithm over 5,000 iterations. The resulting layout achieves a 22.64% improvement over the standard T9 layout.
A Hidden Markov Model resolves the ambiguity between words that share the same gesture code sequence.
- Hidden states: the words the user intends to type
- Observed events: the numerical gesture codes produced by the hand tracker
At each word boundary, the Viterbi algorithm evaluates:
- Emission probability: how well the word matches the typed code
- Transition probability: how likely the word is to follow the previous word, drawn from a 10,000 × 10,000 bigram matrix
Weak paths are continuously pruned, keeping memory usage flat while allowing later words to retroactively correct earlier ambiguities. The decoder also supports prefix-based autocomplete and cross-word-boundary backspace.
Transformers and LSTMs offer strong contextual prediction but require large datasets, high memory, and introduce latency. For a real-time, edge-compatible gesture keyboard, an HMM provides a lightweight, mathematically explainable alternative that executes in microseconds.
Research by Kurtenbach and Buxton (University of Toronto, 1994) found that marking menu accuracy holds well up to 8 items but degrades significantly at 12. Fewer than 8 keys would increase word collisions and strain the prediction model; more than 8 would hurt gesture accuracy. 8 is the sweet spot.
Air-swiping achieves 20–28 WPM (e.g., Vulture). Letter-by-letter air-handwriting struggles to reach half that speed and demands more physical exertion and cognitive load. The directional model is faster for expert users and more accessible for users with motor impairments.
Training data for the HMM spans 100,000+ sentences across formal and informal English:
| Source | Sentences | Character |
|---|---|---|
| Brown University Corpus | 57,300 | Formal grammar, complex sentence structure |
| Webtext Corpus | 10,000 | Conversational, tech-focused language |
| NPS Chat Corpus | 10,500 | Casual slang, messaging shorthand |
| Twitter Samples | 30,000 | Modern, phone-centric vocabulary |
Letter and bigram frequencies from Google Corpus Data are used for keyboard layout scoring.
| Tool | Purpose |
|---|---|
| Python | Core language |
| MediaPipe | Hand landmark detection (21 keypoints) |
| OpenCV | Real-time video capture and frame processing |
| PyAutoGUI | Keyboard and mouse control |
| NumPy | Matrix operations and bigram storage |
| NLTK | Corpus access and natural language processing |
Gesture recognition
- Fast straight strokes occasionally misclassify as arcs (and vice versa)
- Z-axis depth detection for Enter is sensitive to micro-movements
- A 1-second cooldown between gestures limits maximum throughput
Language model
- Fixed vocabulary of the 10,000 most common English words
- Stateless across sessions — context resets on Enter or application exit
- Cannot distinguish between typos and genuine new vocabulary without a verification layer
- Replace fixed pixel thresholds with adaptive filtering (e.g., Kalman filter) for smoother gesture isolation
- Train a lightweight classifier (SVM or MLP) on Rubine features to replace hardcoded geometric thresholds
- Add auditory or haptic feedback to reduce reliance on visual confirmation
- Implement an out-of-vocabulary verification engine for safe vocabulary expansion
- Develop dynamic local adaptation so the model updates with user-specific word frequency over time
- Wobbrock, J. O., Myers, B. A., & Kembel, J. A. — EdgeWrite: A Stylus-Based Text Entry Method Designed for High Accuracy and Stability of Motion
- Kurtenbach, G. & Buxton, W. — The Limits of Expert Performance Using Hierarchic Marking Menus (University of Toronto, 1994)
- Rubine, D. H. — The Automatic Recognition of Gestures (CMU thesis)
- Jurafsky, D. & Martin, J. H. — Speech and Language Processing, Appendix A: Hidden Markov Models