A full-stack market intelligence platform that harvests real-time global news, structures it via an automated ETL pipeline, and surfaces insights through a RAG-powered AI assistant and an interactive Streamlit dashboard.
AI News Intelligence is a business intelligence application built around a three-stage architecture: ingest → analyse → surface. It continuously harvests US headlines from NewsAPI, cleans and deduplicates them into a SQLite database, and then exposes them through two AI-powered engines:
- RAG Assistant — a senior-analyst LLM agent (
gemini-2.5-flash) that answers natural-language queries grounded exclusively in historical database context, with strict hallucination safeguards. - Delta Engine — a narrative drift detector that classifies topics as emerging, persistent, or fading by comparing any two point-in-time report snapshots.
All insights are surfaced through a premium, custom-styled Streamlit dashboard with interactive charts, metric cards, and topic frequency timelines.
In an era of constant news cycles, staying informed without drowning in noise is a genuine challenge — especially for professionals who need to act on information, not just consume it.
| Pain Point | Reality Without This Tool |
|---|---|
| 📰 Information overload | Dozens of sources publish hundreds of articles daily. Reading everything is impossible. |
| ⏳ Time-consuming research | Manually scanning headlines, cross-referencing sources, and spotting trends takes hours. |
| 🧩 No narrative memory | It's hard to recall what topics dominated last week vs. today — context gets lost. |
| 🔍 No natural-language access | Standard news platforms don't let you ask questions about what they've reported. |
| 📉 Missed trend signals | Slow-building or fading stories fly under the radar without systematic tracking. |
AI News Intelligence is built for analysts, researchers, students, and curious professionals who want a single, intelligent platform to:
- ✅ Save hours of daily reading — the ETL pipeline collects and deduplicates headlines automatically, so you only see what's new and relevant.
- ✅ Get the big picture instantly — AI-generated summaries distil dozens of articles into concise, actionable intelligence in seconds.
- ✅ Ask questions in plain English — instead of searching through articles, simply ask "What has been the biggest economic story this week?" and get a grounded answer drawn from real stored data.
- ✅ Track how narratives evolve over time — the Delta Engine automatically flags which topics are rising, holding steady, or disappearing between any two moments in history, giving you an early-warning system for shifting conversations.
- ✅ Trust the answers you get — unlike generic chatbots, the RAG engine is strictly prohibited from speculating beyond what's actually in the database, eliminating hallucination risk.
In short: this tool turns a chaotic news feed into a structured, queryable, time-aware intelligence archive — so you stay informed with minimal effort and maximum confidence.
| Feature | Description |
|---|---|
| 🔄 Automated ETL Pipeline | Fetches, sanitises, deduplicates, and loads raw headlines into SQLite on demand |
| 🔍 Vector Semantic Search | NumPy cosine-similarity retrieval against Gemini gemini-embedding-001 embeddings |
| 🤖 RAG Intelligence Engine | Context-grounded LLM responses with no hallucination beyond stored data |
| 📊 Delta Engine | Point-in-time report comparison to detect emerging, persistent, and fading narratives |
| 📈 Interactive Dashboard | Streamlit UI with topic trends, metric cards, and filterable report history |
| 🛡️ Resilient API Handling | Quota guards and safety logic for high-volume Gemini API calls |
┌─────────────────────────────────────────────────────────────────────────┐
│ AI News Intelligence │
└────────────────────────────────┬────────────────────────────────────────┘
│
┌───────────────────────▼────────────────────────┐
│ ETL Pipeline (app/pipeline.py) │
│ │
│ [NewsAPI] ──► fetch_news ──► clean_data ──► SQLite DB │
└───────────────────────┬────────────────────────┘
│
┌──────────────────┼──────────────────┐
│ │ │
┌─────────▼───────┐ ┌───────▼────────┐ ┌──────▼────────────┐
│ Embeddings Engine│ │ Report Builder│ │ Delta Engine │
│ (embeddings.py) │ │(report_builder)│ │ (report_compare) │
│ gemini-embed-001 │ │ gemini-2.5- │ │ Topic drift & │
│ + NumPy cosine │ │ flash + topics │ │ narrative shifts │
└─────────┬───────┘ └───────┬────────┘ └──────┬────────────┘
│ │ │
└──────────────────┼──────────────────┘
│
┌────────────▼──────────────┐
│ RAG Assistant (rag.py) │
│ gemini-2.5-flash · context │
│ grounded · no hallucination│
└────────────┬───────────────┘
│
┌────────────▼──────────────┐
│ Streamlit Dashboard │
│ (frontend/dashboard.py) │
│ Charts · Metrics · Search │
└───────────────────────────┘
| Layer | Technology | Purpose |
|---|---|---|
| AI / LLM | Google Gemini gemini-2.5-flash |
Report generation, RAG responses |
| Embeddings | Google gemini-embedding-001 + REST |
Vector creation, semantic memory |
| Vector Search | NumPy (cosine similarity) | Retrieval against stored embeddings |
| Data Ingestion | NewsAPI + requests |
Live US headline extraction |
| Data Processing | Pandas, Python re |
Cleaning, deduplication, normalisation |
| Database | SQLite3 | Persistent storage with safe schema migrations |
| Frontend | Streamlit + Custom CSS | Interactive dashboard and analytics UI |
| Config & Secrets | python-dotenv |
Environment variable isolation |
| Logging | Python logging |
Structured runtime diagnostics |
ai-news-intelligence/
├── app/
│ ├── main.py # CLI entry point for manual pipeline runs
│ ├── pipeline.py # ETL orchestrator (Extract → Transform → Load)
│ ├── fetch_news.py # NewsAPI integration
│ ├── clean_data.py # Article sanitisation & deduplication
│ ├── database.py # SQLite schema, migrations, and CRUD operations
│ ├── embeddings.py # Gemini embedding generation & storage
│ ├── semantic_search.py # Cosine similarity retrieval
│ ├── rag.py # RAG response generation (Gemini)
│ ├── ai_summary.py # AI-powered news summary engine
│ ├── report_builder.py # Full intelligence report assembly
│ ├── report_compare.py # Delta Engine — narrative drift detection
│ ├── topic_analytics.py # Topic frequency & trend analysis
│ ├── queries.py # All database query functions
│ ├── config.py # Environment variable loading
│ └── logger.py # Logger factory
├── frontend/
│ └── dashboard.py # Streamlit dashboard (all UI logic)
├── data/
│ └── news.db # SQLite database (auto-created)
├── logs/ # Runtime log files
├── .streamlit/ # Streamlit theme configuration
├── requirements.txt
└── .env # API keys (not committed)
- Python 3.11+
- A NewsAPI API key (free tier available)
- A Google Gemini API key
git clone https://github.com/your-username/ai-news-intelligence.git
cd ai-news-intelligencepython -m venv venv
source venv/bin/activate # macOS / Linux
# venv\Scripts\activate # Windowspip install -r requirements.txtCreate a .env file in the project root:
NEWS_API_KEY=your_newsapi_key_here
GEMINI_API_KEY=your_gemini_api_key_herepython -m app.mainExpected output:
Starting manual pipeline execution...
✅ Success! Fetched: 38 | Cleaned: 35 | Newly Inserted: 35
streamlit run frontend/dashboard.pyNavigate to http://localhost:8501 in your browser.
Coordinates the full ingestion cycle: initialises the database schema, fetches raw articles from NewsAPI, applies cleaning and deduplication, and inserts new records. Returns execution metrics (raw_count, cleaned_count, inserted_count).
Takes a user query and a list of semantically retrieved articles, constructs a strict prompt that forbids the LLM from reasoning beyond the provided context, and returns a grounded, citation-safe answer via gemini-2.5-flash.
Calls the gemini-embedding-001 REST endpoint to generate 768-dimensional embeddings for each stored summary, enabling persistent semantic memory independent of the Gemini Python SDK.
Parses two historical report snapshots and computes the narrative drift: which topics/sources are newly added, which have disappeared, and which persist — enabling trend-over-time business analysis.
A ~14k-line premium UI featuring source filters, date range selectors, topic frequency timelines, a live RAG chat interface, and a Delta Engine comparison panel — all injected with custom CSS for a polished look.
-- Raw ingested headlines
CREATE TABLE articles (
id INTEGER PRIMARY KEY AUTOINCREMENT,
title TEXT NOT NULL,
source TEXT,
published_at TEXT,
url TEXT UNIQUE NOT NULL
);
-- AI-generated summaries with topic metadata and embeddings
CREATE TABLE summaries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
summary_text TEXT NOT NULL,
topics TEXT, -- structured topic:count pairs
source_filter TEXT,
article_limit INTEGER,
embedding TEXT, -- serialised float vector (JSON)
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Full intelligence reports (point-in-time snapshots)
CREATE TABLE reports (
id INTEGER PRIMARY KEY AUTOINCREMENT,
report_timestamp TEXT,
total_articles INTEGER,
displayed_articles INTEGER,
what_matters_now TEXT,
summary_text TEXT,
dominant_topics TEXT,
top_sources TEXT,
source_filter TEXT,
article_limit INTEGER,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);- API keys are loaded exclusively from
.envviapython-dotenvand are never hardcoded. .envis listed in.gitignore— do not commit it.- The RAG prompt enforces strict context boundaries to prevent the LLM from disclosing information outside the stored dataset.
This project is licensed under the MIT License.