Skip to content

Keshav0781/media-intelligence-agent

BranchesTags

Repository files navigation

Media Intelligence Agent

Multimodal AI pipeline for automated video analysis, transcription, and semantic archive search — built for broadcast media and publishing industries.

Python License Tests

🌐 Live Deployment

Service URL
Streamlit UI http://34.107.34.179:8501
FastAPI API http://34.107.34.179:8000
API Docs http://34.107.34.179:8000/docs

Deployed on GCP VM — europe-west3 (Frankfurt, Germany)

Using the Live App

Step 1 — Open the URL Visit http://34.107.34.179:8501 in your browser. First load takes 20-30 seconds — models are loading on the server.

Step 2 — Upload a video

  • Click Process Video in the sidebar
  • Click Upload and select a video from your laptop
  • Recommended: under 7 minutes, under 200MB
  • Supported formats: MP4, MKV, AVI, MOV, WEBM
  • Click ▶ Process Video
  • Processing takes 1-2 minutes (transcription + analysis + indexing)

Step 3 — View results After processing you will see:

  • Language detected (DE, EN, etc.)
  • Broadcast type (news bulletin, interview, etc.)
  • Full summary in English
  • Main topics covered
  • Key stories with exact timestamps

Step 4 — Search the archive

  • Click Search Archive in the sidebar
  • Your processed video is automatically selected
  • Type any question in natural language or keywords
  • Works in any language — English query finds German content
  • Press Enter or click 🔍 Search
  • You get a direct 🤖 Answer (2-3 sentences) + 📚 Sources with relevance scores and timestamps

Step 5 — Switch between videos

  • Previously processed videos appear in Processed Videos section in sidebar
  • Click any video to switch — search results update automatically
  • Video history persists across browser sessions

Important notes:

  • Answer generation uses Gemini API — limited to 20 calls/day on free tier
  • If answer shows "API quota exceeded" — sources are still shown correctly
  • Dangerous or off-topic queries are blocked by guardrails automatically
  • All data persists on server — no need to re-upload previously processed videos

Overview

Media Intelligence Agent transforms raw broadcast video into structured, searchable knowledge. It combines speech recognition, computer vision, and large language models to automatically:

  • Transcribe speech from any video in 99 languages
  • Extract meaningful keyframes using multimodal scene detection
  • Generate editorial summaries and identify key stories with timestamps
  • Index all content for semantic search — supporting both natural language and keyword queries
  • Block harmful or irrelevant queries using local semantic guardrails
  • Expose everything via a production REST API and Streamlit UI

Tested on real Tagesschau (ARD) broadcasts — correctly identifies German language, topics and key stories with precise timestamps.

Pipeline

Video Upload (from browser or API)
    │
    ├── FFmpeg ──────────────► Audio (WAV 16kHz mono)
    │                               │
    │                          Whisper ASR (local)
    │                          - 99 languages auto-detected
    │                          - Timestamped segments
    │                               │
    ├── OpenCV ─────────────► Keyframes (JPG)
    │   Multimodal detection:       │
    │   - Visual: HSV histogram     │
    │   - Audio: speech boundaries  │
    │                               │
    │                      Gemini 3 Flash Preview
    │                      Multimodal analysis:
    │                      - Overall summary
    │                      - Main topics
    │                      - Key stories + timestamps
    │                      - Broadcast type detection
    │                      - MD5 caching (instant on repeat)
    │                               │
    │                      Qdrant Hybrid Search Index
    │                      - Dense vectors (semantic, 384-dim)
    │                      - Sparse vectors (BM25 keyword)
    │                      - RRF fusion
    │                      - CrossEncoder reranking
    │                      - Multilingual search
    │                               │
    │                      LangGraph Orchestration
    │                      - 5 nodes with conditional edges
    │                      - Retry logic (3 attempts)
    │                      - Error handling + graceful stop
    │                      - LangSmith tracing
    │                               │
    │              ┌────────────────┴────────────────┐
    │              │                                 │
    │        FastAPI REST API              Streamlit UI
    │        - POST /pipeline/upload       - Upload video
    │        - POST /pipeline/process      - View results
    │        - POST /search                - Search archive
    │        - GET  /health                - Video history
    │        - Guardrails                  - Answer generation
    │        - Rate limiting               - Switch videos
    │        - Input validation

Components

Component File Status
Audio Extraction app/pipeline/audio_extractor.py ✅ Tested
Speech Recognition app/pipeline/transcriber.py ✅ Tested
Frame Extraction app/pipeline/frame_extractor.py ✅ Tested
Video Analysis app/pipeline/gemini_analyser.py ✅ Tested
RAG Indexer app/rag/indexer.py ✅ Tested
RAGAS Evaluator app/rag/evaluator.py ✅ Tested
Pipeline State app/agents/state.py ✅ Tested
Pipeline Nodes app/agents/nodes.py ✅ Tested
Pipeline Graph app/agents/graph.py ✅ Tested
Pipeline Agent app/agents/pipeline_agent.py ✅ Tested
FastAPI REST API app/api/main.py ✅ Tested
API Routes (health, pipeline, search) app/api/routes/ ✅ Tested
Semantic Guardrails app/api/guardrails.py ✅ Tested
Streamlit UI app/ui/main.py ✅ Tested

Key Features

Multimodal Scene Detection Combines HSV histogram comparison with Whisper speech boundaries for keyframe extraction. In Tagesschau broadcasts — 84% of keyframes confirmed by both visual and audio signals.

Adaptive Processing

  • Videos < 10 minutes → single Gemini call (faster, cheaper)
  • Videos ≥ 10 minutes → chunked parallel processing + hierarchical summarisation
  • MD5 caching — repeated video analysis returns instantly (0.04s)

Two-Stage Hybrid Search Stage 1: Hybrid retrieval — BM25 sparse + dense semantic vectors combined with RRF fusion — retrieves top 20 candidates. Stage 2: CrossEncoder reranking — reads query and document together for precise relevance scoring — selects top results.

Handles:

  • Natural language: "minister who lied about toll road"
  • Keywords: "Scheuer PKW Maut"
  • Cross-language: English query → German content ✅

LangGraph Orchestration

  • 5 nodes connected with conditional edges
  • Retry logic — transient failures retried 3 times automatically
  • Error handling — pipeline stops gracefully on failure
  • LangSmith tracing — every node execution tracked with timing

Local Semantic Guardrails

  • Uses existing sentence-transformers model — zero API calls
  • Blocks harmful queries (weapons, hacking, illegal content)
  • Blocks prompt injection attempts
  • Returns clear user-friendly error messages
  • Works 24/7 regardless of API quota status

Streamlit UI

  • Upload any video from laptop directly
  • Process video — see summary, topics, stories with timestamps
  • Search archive with natural language or keywords
  • Generated answers using Gemini (2-3 sentences, based only on video content)
  • Video history in sidebar — switch between multiple processed videos
  • Persistent history — survives browser refresh

Production REST API

  • POST /api/pipeline/upload — upload and process video file
  • POST /api/pipeline/process — process video by file path
  • POST /api/search — search indexed archive
  • GET /api/health — health check for monitoring
  • Input validation — 422 for invalid requests
  • Rate limiting — 10 requests per minute per IP
  • OpenAPI docs at /docs

RAGAS Evaluation

Evaluated on real Tagesschau (ARD) German broadcast content using RAGAS framework.

Metric Score Meaning
Faithfulness 1.000 Zero hallucination — answer contains only retrieved content ✅
Context Precision 1.000 All retrieved chunks are relevant to the query ✅
Context Recall 1.000 All relevant content is retrieved — nothing missed ✅
Answer Relevancy 0.808 Answer is relevant to question ✅

How scores improved

Faithfulness: 1.000 Answer is built directly from retrieved content — Gemini adds nothing from its own training data. Zero hallucination. Most critical metric for a media archive system where accuracy is essential.

Context Precision: 1.000 — improved from 0.887 Before CrossEncoder reranking — some irrelevant documents appeared in top results. After CrossEncoder two-stage retrieval — all returned documents are genuinely relevant.

Context Recall: 1.000 — improved from 0.500 Before CrossEncoder — hybrid search returned top 5 candidates directly. Relevant content ranked 6th-10th was never seen. After CrossEncoder — system fetches top 20 candidates first, then reranks — nothing relevant is missed.

Answer Relevancy: 0.808 Answer generated by Gemini from retrieved context — focused 2-3 sentence response directly addressing the question. Improved from 0.637 (raw context) to 0.808 (generated answer).

Tech Stack

Category Technology
Speech Recognition OpenAI Whisper (local, base model)
Multimodal LLM Google Gemini 3 Flash Preview
Vector Database Qdrant (local mode)
Embeddings paraphrase-multilingual-MiniLM-L12-v2 (384 dim)
Keyword Search BM25 (rank-bm25, persisted to disk)
Reranking CrossEncoder ms-marco-TinyBERT-L-2-v2
Agent Orchestration LangGraph
Observability LangSmith
REST API FastAPI + uvicorn
UI Streamlit
Guardrails Local sentence-transformers (zero API calls)
Evaluation RAGAS
Rate Limiting slowapi
Testing pytest (75 tests)
Deployment GCP VM — europe-west3 Frankfurt
Process Manager Supervisor (auto-restart)
Containerisation Docker (Dockerfile included)

Getting Started

Prerequisites

Installation

git clone https://github.com/Keshav0781/media-intelligence-agent.git
cd media-intelligence-agent
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
# Add your API keys to .env

Environment Variables

GEMINI_API_KEY=your_gemini_api_key
GEMINI_MODEL=models/gemini-3-flash-preview
LANGSMITH_API_KEY=your_langsmith_api_key
LANGSMITH_PROJECT=media-intelligence-agent
LANGCHAIN_TRACING_V2=true

Run Tests

# Run all tests except Gemini analyser (saves API quota)
python -m pytest -v --ignore=app/pipeline/test_gemini_analyser.py

# Run all tests including Gemini analyser
python -m pytest -v

Start Services

# Terminal 1 — API server
python -m uvicorn app.api.main:app --host 0.0.0.0 --port 8000 --reload

# Terminal 2 — Streamlit UI
streamlit run app/ui/main.py

Process a Video via API

# Upload and process a video file
curl -X POST http://localhost:8000/api/pipeline/upload \
  -F "file=@data/test_video.mp4"

# Process by file path
curl -X POST http://localhost:8000/api/pipeline/process \
  -H "Content-Type: application/json" \
  -d '{"video_path": "data/test_video.mp4"}'

Search the Archive

curl -X POST http://localhost:8000/api/search \
  -H "Content-Type: application/json" \
  -d '{"video_hash": "YOUR_VIDEO_HASH", "query": "What happened with Scheuer?", "limit": 5}'

Project Structure

media-intelligence-agent/
├── app/
│   ├── pipeline/
│   │   ├── audio_extractor.py
│   │   ├── transcriber.py
│   │   ├── frame_extractor.py
│   │   ├── gemini_analyser.py
│   │   ├── test_audio_extractor.py
│   │   ├── test_transcriber.py
│   │   ├── test_frame_extractor.py
│   │   └── test_gemini_analyser.py
│   ├── rag/
│   │   ├── indexer.py
│   │   ├── evaluator.py
│   │   └── test_indexer.py
│   ├── agents/
│   │   ├── state.py
│   │   ├── nodes.py
│   │   ├── graph.py
│   │   ├── pipeline_agent.py
│   │   └── test_pipeline_agent.py
│   ├── api/
│   │   ├── main.py
│   │   ├── guardrails.py
│   │   ├── test_api.py
│   │   ├── routes/
│   │   │   ├── health.py
│   │   │   ├── pipeline.py
│   │   │   └── search.py
│   │   └── models/
│   │       ├── requests.py
│   │       └── responses.py
│   ├── ui/
│   │   └── main.py
│   ├── config.py
│   └── logger.py
├── tests/
│   └── test_pipeline_integration.py
├── config.yaml
├── conftest.py
├── pytest.ini
├── Dockerfile
├── requirements.txt
└── .env.example

Configuration

All settings centralized in config.yaml — no hardcoded values in code:

pipeline:
  frames:
    histogram_threshold: 0.4
    min_interval_seconds: 2.0
  gemini:
    chunk_size_seconds: 30.0
    direct_analysis_threshold: 600.0
rag:
  embedding_model: paraphrase-multilingual-MiniLM-L12-v2
  vector_dim: 384
  reranker_model: cross-encoder/ms-marco-TinyBERT-L-2-v2
  reranker_candidates: 20

API Reference

Health Check

GET /api/health

Process Video (file path)

POST /api/pipeline/process
Body: {"video_path": "data/test_video.mp4"}

Upload and Process Video

POST /api/pipeline/upload
Form: file=<video file>
Max size: 200MB
Supported: .mp4, .mkv, .avi, .mov, .webm

Search Archive

POST /api/search
Body: {
  "video_hash": "03b26d23864237a264d0e262ccbb655c",
  "query": "What happened with Scheuer?",
  "limit": 5,
  "content_type": "story"  // optional: segment, story, summary, topic
}

Author

Keshav Jha — AI Engineer, Erlangen Germany LinkedIn | GitHub

About

Multimodal AI agent for automated video summarisation and archive indexing for media industry

34.107.34.179:8501

Topics

python nlp gemini broadcast whisper rag fastapi streamlit media-intelligence langchain

Resources

Readme

License

MIT license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases 1

Media Intelligence Agent v1.0.0 Latest
Jun 2, 2026

Packages

 
 
 

Contributors

Languages