🎥 Video Intelligence Agent

An AI-powered video understanding application that converts videos into searchable knowledge. The system extracts audio from YouTube or local videos, generates transcripts, produces concise summaries, extracts key insights, and enables semantic question answering using Retrieval-Augmented Generation (RAG).

Instead of manually watching lengthy videos, users can quickly understand the content, identify important decisions or action items, and interact with the video through natural language queries.

---

✨ Features

• 🎥 Process YouTube videos or local video/audio files
• 🎤 Automatic speech-to-text transcription using Whisper and Sarvam AI
• 📝 AI-generated video summaries
• 🏷 Automatic title generation
• ✅ Extract Action Items
• 📌 Extract Key Decisions
• ❓ Identify Open Questions
• 🔍 Chat with videos using Retrieval-Augmented Generation (RAG)
• ⚡ Interactive Streamlit interface

---

Tech Stack

• Python
• Streamlit
• LangChain
• Mistral AI
• Whisper
• Sarvam AI
• ChromaDB
• HuggingFace Embeddings
• Retrieval-Augmented Generation (RAG)
• yt-dlp
• PyDub

---

Architecture

Video / YouTube URL
        │
        ▼
Audio Extraction (yt-dlp / PyDub)
        │
        ▼
Speech Transcription
(Whisper / Sarvam AI)
        │
        ▼
Transcript
        │
 ┌──────┼──────────────┐
 ▼      ▼              ▼
Summary  Insights      Vector Database
                      (ChromaDB)
                           │
                           ▼
                    Semantic Retrieval
                           │
                           ▼
                      Ask Questions

---

Project Structure

.
├── core
│   ├── extractor.py          # Action items, decisions & questions
│   ├── rag_engine.py         # RAG pipeline
│   ├── summarizer.py         # Summary & title generation
│   ├── transcriber.py        # Whisper & Sarvam transcription
│   └── vector_store.py       # ChromaDB vector store
│
├── utils
│   └── audio_processor.py    # Audio extraction & chunking
│
├── app.py                    # Streamlit UI
├── main.py                   # Application entry point
├── test.py
├── requirements.txt
└── .gitignore

---

Workflow

1. Input

Upload

• Local Video
• Local Audio
• YouTube URL

---

2. Audio Processing

The application

• downloads YouTube audio
• converts media to WAV
• normalizes audio
• chunks long recordings

---

3. Speech Transcription

Supports two transcription engines

Whisper

• Local inference
• English transcription

Sarvam AI

• Hinglish speech recognition
• English translation

---

4. AI Video Understanding

Using LangChain + Mistral AI

• Generate concise summaries
• Generate video titles
• Extract action items
• Extract decisions
• Detect unanswered questions

---

5. Video Chat (RAG)

The transcript is

• chunked
• embedded using HuggingFace embeddings
• indexed in ChromaDB

User questions are answered using only the retrieved transcript context, improving factual accuracy and reducing hallucinations.

---

Installation

git clone https://github.com/AmanXk/video-agent.git

cd video-agent

Create virtual environment

python -m venv .venv

Activate

Windows

.venv\Scripts\activate

Linux/macOS

source .venv/bin/activate

Install dependencies

pip install -r requirements.txt

---

Environment Variables

Create a .env

MISTRAL_API_KEY=your_mistral_api_key

SARVAM_API_KEY=your_sarvam_api_key

WHISPER_MODEL=small

SARVAM_STT_MODEL=saaras:v2.5

---

Run

streamlit run app.py

---

Supported Inputs

• MP4
• MP3
• WAV
• AVI
• MOV
• YouTube URLs

---

AI Capabilities

• Video Transcription
• AI Summarization
• Information Extraction
• Retrieval-Augmented Generation
• Semantic Search
• Natural Language Question Answering

---

Future Improvements

• Speaker diarization
• Timeline-based video navigation
• Multi-language transcription
• PDF summary export
• Video chapter generation
• Cloud deployment

---

Author

Aman Kumar

If you found this project useful, consider giving it a ⭐ on GitHub