Stream analytics monitor for Wikipedia change events implementing the DGIM algorithm from scratch — EECS4415 Big Data, York University
A Python-based stream analytics monitor that tracks Wikipedia edit activity by top users and profiles their behaviour using the DGIM (Datar-Gionis-Indyk-Motwani) Algorithm — a memory-efficient technique for approximating counts over data streams.
Built as the course project for EECS4415 (Big Data) at York University, Winter 2025.
Reads a stream of change events from the Wikimedia platform and builds a lightweight statistical profile for each tracked user — capturing when they were active and how frequently — without storing the full event history.
The system has three components:
1. Stream Ingestion and Synopsis Building
- Implements the DGIM algorithm from scratch to incrementally build a per-user synopsis as change events are processed
- Treats the full stream as a single window (no sliding window required)
- Exposes query functions to retrieve approximate event counts across time intervals
2. User Profiling and Accuracy Analysis
- Generates hourly histograms of user edit activity using DGIM approximations
- Plots absolute error of predicted histograms against ground truth counts
- Plots relative error of DGIM estimation versus ground truth across time
- Investigates how the R parameter (bucket count) affects approximation accuracy
3. Findings and Analysis
- Documents observations on DGIM's strengths and limitations across users with different activity frequencies
- Explores how these techniques could be extended toward bot detection
- DGIM approximation accurately captured overall user activity trends despite its lightweight memory footprint
- Higher R values (more buckets) significantly reduced relative error, confirming the expected precision/memory trade-off
- High-frequency users produced larger approximation errors due to aggressive bucket merging — DGIM performs best on low-to-moderate activity profiles
- Consistent high-frequency edit patterns with low hourly variance emerged as a potential signal for bot detection
| Component | Technology |
|---|---|
| Language | Python 3 |
| Algorithm | DGIM (implemented from scratch) |
| Data Processing | Python standard library, custom stream parser |
| Visualization | Matplotlib |
| Environment | Google Colab / Jupyter Notebook |
WikiMon_EECS4415N.ipynb # Full notebook with implementation, analysis, and report
- Open the notebook in Google Colab or Jupyter
- Run all cells in order from top to bottom
- Output cells display histograms, error plots, and analysis results inline
No external dependencies beyond standard Python scientific libraries (matplotlib, collections).
Course project for EECS4415 — Big Data, York University (Winter 2025). Notebook template provided by Prof. Parke Godfrey and Prof. Aijun An. DGIM algorithm implementation and all analysis written independently.