Sign Language Detection Engine

Overview

This repository presents an integrated real-time Sign Language Recognition system supporting both Indian Sign Language (ISL) and American Sign Language (ASL). The system combines the Ultralytics YOLOv8 object detection framework, Google MediaPipe Hands, and a FastAPI-powered web application to provide an end-to-end sign language detection and recognition pipeline.

The system is designed to detect and classify hand gestures from both Indian and American Sign Language alphabets, enabling users to perform sign gestures through images, uploaded media, or live video streams. MediaPipe Hands is used for robust hand detection and landmark estimation, helping localize hand regions and improve recognition stability, while YOLOv8 performs gesture detection and classification.

The project is implemented as an integrated web-based application using FastAPI, connecting the computer vision models with a real-time user interface. The architecture supports multiple recognition workflows within a single platform, allowing users to select the appropriate sign language and perform real-time inference. This combination of YOLOv8 + MediaPipe + FastAPI provides a scalable foundation for sign language recognition across different hand orientations, backgrounds, and lighting conditions.

The repository also includes the complete workflow for dataset preparation, preprocessing, visualization, model training, evaluation, and inference for both ISL and ASL recognition. Model development and training were performed using GPU-accelerated environments, while the trained models are integrated into the FastAPI application for real-time deployment and inference.

---

🇺🇸 American Sign Language (ASL) Model

For real-time American Sign Language (ASL) recognition, our system uses the YOLOv8 object detection framework integrated with computer vision-based hand gesture processing. The model detects and classifies hand gestures representing ASL alphabet signs from live video streams. MediaPipe Hands is used for robust hand detection and landmark tracking, improving stability across different hand positions, orientations, and backgrounds. The trained model is integrated into the FastAPI web application for real-time inference and visualization.

Kaggle Notebook: ASL: https://www.kaggle.com/code/sayaksamanta/sign-language-detection

American Sign Language	American Sign Language Detection

🇮🇳 Indian Sign Language (ISL) Model

To enable real-time interpretation of Indian Sign Language (ISL), our system adopts a Hybrid CNN–Transformer Detection Framework that combines convolutional feature extraction with Transformer-based attention mechanisms for robust spatial and contextual representation of hand gestures.

The model is designed to improve recognition across variations in hand orientation, background, lighting conditions, and gesture appearance, making it suitable for real-time sign language interpretation.

Kaggle Notebook: ISL: https://www.kaggle.com/code/sayaksamanta/indian-sign-language-detction/notebook

Detailed ISL Model Architecture & Implementation: View the Indian Sign Language Model Repository

---

Getting Started

# Clone the repository
git clone https://github.com/say217/Sign-Language-Detection-Engine-.git
cd Sign-Language-Detection-Engine-

# Create virtual environment
python -m venv venv

# Activate virtual environment — Windows
venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Run FastAPI application
uvicorn src.Main.run:app --reload

---

Roadmap: Future Work

The current system recognizes isolated signs frame-by-frame. The next phase of this project moves from isolated gesture classification toward a continuous, conversational sign-language interpreter that can plug into real communication settings such as video calls. Three tracks are planned:

1. Live Video-Call Integration (Zoom / Meet / Teams)

Goal: let a Deaf or hard-of-hearing participant sign naturally during a video call and have captions or translated speech appear for other participants in real time, without either side needing a separate device.

Planned approach:

• Virtual camera / plugin bridge — route the detection engine's annotated (or clean) video feed through a virtual camera device (e.g. via pyvirtualcam or OBS Virtual Camera on the client side), so any conferencing tool — Zoom, Google Meet, Microsoft Teams — can select it as a webcam source without needing native platform integration.
• Zoom Apps SDK track — in parallel, evaluate building a proper Zoom App (using the Zoom Apps SDK / Meeting SDK) that runs the detection pipeline against the meeting's live video track and renders captions as an in-meeting overlay, which is a cleaner experience than a virtual camera but requires Zoom's app review process.
• Low-latency streaming — the current MJPEG-over-HTTP approach used for the local dashboard is not suitable for this use case; call integration will require a lower-latency transport (e.g. WebRTC data channel for caption text, since only text/audio needs to reach the call, not annotated video).
• Bandwidth-aware inference — detection will need to run client-side or on an edge/edge-adjacent server rather than round-tripping full video frames to a central server, to keep caption latency low enough to feel conversational.

2. Transformer-Based Sentence Formation

Goal: move from "classify this single frame as one gloss/letter" to "read a sequence of signs and produce a grammatically correct sentence" — the same gap that exists between OCR-ing individual characters and actually reading a sentence.

Planned approach:

• Sequence modeling over landmark streams — instead of classifying each frame independently, feed a rolling window of MediaPipe landmark sequences into a Transformer encoder (or a Conformer/temporal-attention hybrid) so the model can use motion and context, not just a static hand pose, to disambiguate signs that look similar in isolation.
• Gloss-to-text translation layer — ISL and ASL grammar do not map word-for-word onto spoken-language grammar (e.g. topic-comment ordering, non-manual markers like facial expression carrying grammatical meaning). A second-stage sequence-to-sequence Transformer (encoder-decoder, similar in spirit to a small NMT model) is planned to translate the recognized gloss sequence into fluent sentences in the target spoken language, rather than emitting a literal word-for-word gloss string.
• Incorporating non-manual signals — longer-term, incorporating facial landmark and head-pose streams (also obtainable from MediaPipe) alongside hand landmarks, since grammar in both ISL and ASL is partly carried by eyebrow position, mouth shape, and head tilt, not hand shape alone.
• Streaming decode — the sentence-formation model will need to support incremental/streaming decoding (emitting partial sentences as signing continues) rather than waiting for a full utterance to end, to keep the video-call use case responsive.

3. Audio Translation Output

Goal: once a fluent sentence has been produced, speak it aloud (or into the call's audio channel) in the listener's preferred spoken language, and — for the reverse direction — transcribe the hearing participant's speech so a Deaf participant can read it.

Planned approach:

• Text-to-speech (TTS) output — feed the Transformer's sentence output into a TTS engine to generate spoken audio, injected into the call as a virtual microphone (mirroring the virtual-camera approach above), so remote participants hear a translated voice rather than reading captions.
• Multilingual translation before synthesis — insert a translation step (e.g. a machine translation model or API) between gloss-to-text output and TTS so the spoken output language can differ from the sign language's "home" spoken language (for example, ISL gloss to spoken Hindi or English, ASL gloss to Spanish, etc.), based on the listener's preference.
• Reverse channel: speech-to-sign context — for two-way communication, transcribe hearing participants' speech (via an ASR model) and surface it as text/captions to the Deaf participant, which is the more tractable direction of the two, but still needs to be latency-matched with the sign-to-speech direction so the conversation doesn't feel lopsided.
• Voice/latency trade-offs — evaluate lightweight on-device TTS/ASR models versus higher-quality cloud APIs, since call integration puts a hard ceiling on acceptable round-trip latency.

Why this order

Sentence formation is the hardest and most foundational piece — good video-call integration or audio output is not very useful if the underlying output is still isolated glosses rather than coherent sentences. The current plan is: harden continuous sign recognition and gloss-to-sentence translation first, then layer audio output on top of a working text pipeline, and treat the video-call integration as the final "packaging" step once the recognition-to-sentence pipeline is reliable enough to run continuously rather than on demand.