This repository contains a comprehensive system for analyzing political news coverage through automated trend detection and sentiment analysis. The pipeline processes thousands of news articles daily, identifies emerging narratives, tracks their evolution over time, and provides bias-corrected sentiment analysis—all designed to cut through the noise of modern political media.
- Intelligent Deduplication: Filters out redundant wire service republications while preserving unique perspectives
- Semantic Clustering: Groups related stories using hybrid semantic and entity-based similarity
- Trend Persistence: Tracks evolving narratives across days and weeks, not just isolated daily snapshots
- Bias Correction: Adjusts sentiment scores based on source-specific baselines to reveal relative changes
- Interactive Visualization: Streamlit dashboard for exploring trends, sentiment trajectories, and article distributions
The system operates as a linear processing pipeline with evolutionary feedback, designed to handle the high-velocity, high-redundancy nature of political news.
Raw CSV Data → Enrichment → Deduplication → Daily Clustering → Trend Tracking → Narrative Synthesis → Sentiment Analysis → Visualization
Module: processor.py
This is the entry point for raw data ingestion. It performs two critical transformations through Named Entity Recognition (NER) using spaCy to extract people, organizations, and locations from article text, applying a stop_entities filter to exclude generic political noise like "Biden", "GOP", or "White House" that would otherwise cause over-clustering around common figures rather than specific events. The module also handles embedding generation, converting article titles and descriptions into 768-dimensional semantic vectors using the BAAI/bge-base-en-v1.5 model, allowing the system to recognize that "White House announces layoffs" and "Federal workforce reduction in Washington" describe the same underlying story.
Output: Enriched dataset with embeddings and filtered entity lists.
Module: deduplication.py
News outlets frequently republish identical wire service stories from AP, Reuters, and other agencies. This module uses greedy cosine similarity matching with a threshold of approximately 0.90 to identify near-duplicates and retain only the earliest or most authoritative version. This typically reduces dataset size by 30-50% while preserving narrative diversity.
Modules: clustering.py, tracker.py, trend_manager.py
Instead of batch-processing the entire dataset, the system analyzes one day at a time to model how stories emerge and evolve. The daily clustering component uses Agglomerative Clustering with a hybrid distance metric: 60% semantic similarity based on cosine distance between embeddings, and 40% entity overlap using Jaccard similarity of extracted entities. A semantic override ensures that articles with greater than 0.85 semantic similarity are forced into the same cluster regardless of entity mismatch.
The trend management layer checks if today's clusters are continuations of yesterday's trends, with gap tolerance allowing trends to persist across up to 3 days of silence before being closed. The merging logic identifies similar trends within a rolling window and combines them to prevent fragmentation, ultimately creating multi-day narrative threads with assigned article IDs.
Module: narrative.py
This component transforms raw clusters into human-readable summaries by aggregating headlines within each trend and using LLMs like GPT-4o-mini or heuristic methods to create concise, descriptive titles and one-sentence summaries of the narrative arc.
Module: exporter.py
Generates two research-ready CSV files: a trends timeline providing a high-level view with trend metadata including ID, title, summary, date range, and article count; and an article mappings file offering granular mapping of every article to its assigned Trend ID.
Module: sa_finbert.py
Applies FinBERT, a BERT model fine-tuned for financial and economic text, to perform domain-aware sentiment analysis on both aggregated trend summaries and individual articles. FinBERT provides positive, negative, and neutral probability distributions, serving as a validation layer alongside baseline sentiment scores. Financial news models often perform well on policy and economic news due to training on similar regulatory and market-impact language.
Module: app.py
A Streamlit-based interface for exploring pipeline outputs with several key features. The bias correction mechanism adjusts sentiment scores relative to each source's historical baseline—for example, a source with 80% average negative coverage showing 70% negative sentiment represents a relatively positive shift. Temporal smoothing uses EWMA (Exponentially Weighted Moving Average) to visualize sentiment trends without daily jitter. Users can filter trends by date range, source, sentiment, or keyword, and drill down into any trend to see its constituent articles.
The pipeline begins with raw CSV data flowing into processor.py for NER and embedding generation. The enriched data passes through deduplication.py for similarity filtering before entering tracker.py, which orchestrates a date-by-date processing loop. Each daily slice goes to clustering.py for Agglomerative Clustering, with results flowing to trend_manager.py for trend linking and persistence tracking. The identified trends move to narrative.py for LLM-based summarization, then to exporter.py for CSV generation. The exported data is enriched by sa_finbert.py with sentiment scores before finally being consumed by app.py for interactive visualization.
.
├── processor.py # Data loading and enrichment
├── deduplication.py # Similarity-based deduplication
├── clustering.py # Daily clustering logic
├── tracker.py # Main pipeline orchestrator
├── trend_manager.py # Trend persistence and merging
├── narrative.py # LLM-based summarization
├── exporter.py # CSV output formatting
├── sa_finbert.py # Sentiment analysis
├── app.py # Streamlit dashboard
├── config.yaml # Configuration parameters
├── requirements.txt # Python dependencies
└── data/
├── raw/ # Input CSV files
├── processed/ # Intermediate outputs
└── final/ # Dashboard-ready data
This pipeline was designed for thesis research on media bias and narrative framing in political coverage. It enables tracking how different news sources frame the same events over time, identifying emerging political narratives before they enter mainstream discourse, quantifying sentiment divergence between partisan media ecosystems, and analyzing the lifecycle of political controversies from emergence through peak to decay.
This project includes my master's thesis documenting the research methodology and findings.
📄 master's thesis.pdf
- Real-time news ingestion via RSS and API feeds
- Multi-language support for international news analysis
- Graph-based trend visualization showing narrative connections
- Automated bias report generation
- Integration with fact-checking APIs