A real-time AI-powered mobile application that translates Indian Sign Language (ISL) gestures into text and speech, and converts typed text back into ISL sign representations using video/GIF output.
The AI-Powered Indian Sign Language (ISL) Translator is a final year level engineering project developed by a 2nd year student, designed to improve communication accessibility between deaf or mute individuals and non-sign language users.
The system uses MediaPipe hand landmark detection and TensorFlow Lite on-device inference to recognize ISL gestures in real time through a mobile application built with Flutter. The application also supports reverse translation by converting typed text into ISL sign animations using video or GIF-based outputs.
The project focuses on lightweight mobile deployment, real-time processing, accessibility, and scalable AI architecture.
https://github.com/kashish836/isl-translator
| Category | Technologies |
|---|---|
| Programming Languages | Python, Dart |
| Mobile Framework | Flutter |
| Computer Vision | OpenCV, MediaPipe 0.10.35 |
| Machine Learning | TensorFlow 2.21, Keras, Scikit-learn |
| Mobile AI Inference | TensorFlow Lite (TFLite) |
| Speech Output | Android Text-to-Speech (TTS) |
| Data Processing | NumPy, Pandas, Scikit-learn |
| Visualization | Matplotlib |
- Real-time Indian Sign Language gesture recognition
- Hand landmark detection using MediaPipe
- On-device TensorFlow Lite inference — works fully offline
- Gesture-to-text conversion
- Gesture-to-speech conversion using Android TTS
- Typed text to ISL sign video/GIF translation
- Lightweight mobile-first architecture — model size 68.2 KB
- Modular and scalable AI pipeline
- Flutter-based Android application
- Optimized for real-time performance
Typed Text
↓
Word / Character Mapping
↓
ISL Video or GIF Retrieval
↓
Sign Language Playback in Flutter
| Metric | Result |
|---|---|
| Dataset | Kaggle ISL — 35 classes, 42,745 images |
| Landmarks Extracted | 41,175 samples (96.3% detection rate) |
| Random Forest Accuracy | 99.96% |
| MLP Neural Network Accuracy | 99.89% |
| 5-Fold Cross Validation | 98.85% mean, 1.30% std |
| TFLite Model Size | 68.2 KB |
| Keras Model Size | 262.8 KB |
The controlled evaluation accuracy achieved was approximately 99%, while real-world performance. Note: 99.96% reflects offline test accuracy only. See "Phase 4 — Real-World Results" below for actual on-device performance (~80-82%).
ISL python/
├── data/
│ ├── raw/ ← dataset images (gitignored)
│ ├── landmarks/ ← dataset.csv (gitignored)
│ └── processed/ ← cleaned data
├── models/
│ ├── saved/ ← trained Keras models (gitignored)
│ └── tflite/ ← .tflite files (gitignored)
├── src/
│ ├── extract_landmarks.py ← Phase 2 ✅
│ ├── train_model.py ← Phase 3
│ ├── predict_realtime.py ← Phase 3
│ └── collect_data.py ← Future
├── flutter_app/
│ ├── lib/
│ │ ├── main.dart ✅
│ │ ├── screens/home_screen.dart ✅ full UI + pipeline
│ │ ├── services/
│ │ │ ├── camera_service.dart ✅
│ │ │ ├── tflite_service.dart ✅
│ │ │ └── hand_landmark_service.dart ✅
│ │ └── providers/prediction_provider.dart ✅
│ ├── assets/models/
│ │ ├── isl_model_v3.tflite ✅ 241KB (active)
│ │ ├── labels.txt ✅ 35 classes
│ │ └── hand_landmarker.task ✅ 7.8MB
│ └── android/
│ ├── app/build.gradle.kts ✅
│ └── app/src/main/kotlin/.../
│ └── MainActivity.kt ✅ MediaPipe native
├── PROJECT_CONTEXT.md ← This file
├── ARCHITECTURE.md ← After Phase 3
├── CODING_RULES.md ← Coding standards
├── FEATURE_LOG.md ← Change tracking
├── .gitignore
├── NOTICE
├── README.md
└── requirements.txt
---
## Setup Instructions
### 1. Clone Repository
```bash
git clone https://github.com/kashish836/isl-translator.git
cd isl-translator
py -3.11 -m venv isl_envWindows:
isl_env\Scripts\activatepip install -r requirements.txtpython src/predict_realtime.py| Phase | Description | Status |
|---|---|---|
| Phase 1 | Environment setup | ✅ Complete |
| Phase 2 | Dataset + landmark extraction | ✅ Complete |
| Phase 3 | Model training + TFLite export | ✅ Complete |
| Phase 3.5 | Real-time Python webcam testing | ✅ Complete |
| Pre-Phase 4 | Full documentation suite | ✅ Complete |
| Phase 4 | Flutter app + on-device inference | ✅ Complete |
| Phase 4.5 | UI polish + app logo | ✅ In Progress |
| Phase 5 | Text-to-sign module | ✅ Complete |
| Phase 6 | Testing + report + demo | 🔄 In Progress |
| Level | Target | Status |
|---|---|---|
| Level 1 | Working prototype — college submission | 🔜 In Progress |
| Level 2 | Polished product — Play Store ready | 🔜 Future |
| Level 3 | Commercial version — proprietary data | 🔜 Future |
| Dataset | Source | License |
|---|---|---|
| Indian Sign Language ISL | Kaggle — prathumarikeri | CC BY-SA 4.0 |
Datasets used for academic research and prototype development only. All third-party datasets are credited to their respective authors. Commercial version will use proprietary recorded data.
| Paper | Authors | Year | Journal |
|---|---|---|---|
| An Integrated MediaPipe-Optimized GRU Model for ISL Recognition | Barathi Subramanian et al. | 2022 | Scientific Reports |
| Automatic ISL Recognition using MediaPipe Holistic and LSTM | G Khartheesvar et al. | 2024 | Multimedia Tools and Applications |
| ISL Recognition and Translation using MediaPipe and LSTM | Tanmay Nehra et al. | 2023 | WCCC 2023 |
| Real-time Interpreter for ISL Using MediaPipe and Deep Learning | Suguna Mariappan et al. | 2024 | Information Technology and Control |
| ISL Sign Language Recognition Using LSTM-Driven Deep Learning | Guruprakash B et al. | 2024 | Journal of Electrical Systems |
- No raw dataset stored in repository
- No API keys or passwords in codebase
- All inference runs on-device — no data leaves the phone
- Input validation implemented in Flutter app
- Model obfuscation via ProGuard (Phase 4)
Camera (back) → YUV frames → rotateBitmap(90°)
→ MediaPipe HandLandmarker (Kotlin native)
→ 21 landmarks → normalizeLandmarks() — exact Kotlin port
of Python normalize_landmarks(), no coordinate transform
→ isl_model_v3.tflite (241KB, Kotlin TFLite inference)
→ {label, confidence} → Flutter UI
→ Sentence building → flutter_tts speech output
Note: The original "Project Architecture" diagram above reflects the Phase 3 / early-Phase 4 design. The diagram above is what actually shipped after Phase 4 debugging — inference moved fully to Kotlin, back camera required, and the TFLite model was reconverted (see Results table).
- Single-hand signs only — two-hand ISL signs not supported (documented future work)
- ~80-82% real-world accuracy (vs 99.96% offline test accuracy)
- Accuracy depends on lighting conditions
- Accuracy depends on camera angle and distance
- Some signs share similar landmark geometry (e.g. 1/M, S/6, 2/V)
- Designed for Android devices only (Android 9+)
- Back camera required — front camera support requires an additional mirror-coordinate transform that was not implemented in Level 1 (planned for Level 2)
- Text-to-Sign currently uses static JPG images — video clips planned for Level 2 once own ISL dataset is recorded
- Speech-to-text (STT) accuracy depends on device microphone and system STT engine quality
| Metric | Value |
|---|---|
| Offline Test Accuracy (Random Forest) | 99.96% |
| Cross-Validation Accuracy | 98.85% |
| Real-World Device Accuracy | ~80-82% |
| TFLite Model (final, shipped) | isl_model_v3.tflite (241KB) |
| Coordinate system fix | Back camera + 90° bitmap rotation |
| Inference location | Kotlin native (not Dart) |
| Device Tested | Oppo A12, Android 9, API 28 |
| APK Size | 32MB (arm64 release) |
| Stability threshold (final) | 2 frames, 0.70 confidence |
| Permissions | Camera + Microphone (STT) |
Offline accuracy reflects the held-out test split of the training dataset. Real-world accuracy reflects live testing on a physical device — this is what users actually experience. The original 68.2KB TFLite model (referenced in the ML Model Results table above) could not load on-device in release mode; it was reconverted without optimization to isl_model_v3.tflite (241KB) to fix a TFLite op-version compatibility issue. See FEATURE_LOG.md, June 14–20 entries, for full history.
Kashish Bhiwapurkar 2nd Year Engineering Student
This project is licensed under the MIT License.