🗺️ Google Maps Review Intelligence Platform

Scrape thousands of Google Maps reviews → NLP sentiment & topic analysis → 8-page animated dashboard → Actionable business insights

Bilingual ◆ 2.2M+ reviews across 16 areas ◆ 17,900+ places ◆ 5,100+ lines of Python ◆ 38GB of collected data — and still collecting 24/7

🏗️ Architecture — 24/7 Production Collector

This isn't a one-off script. It runs as a systemd service, 24/7, against a client's business listings — a supervised loop that spins up Docker scraper containers, merges results with DuckDB, and re-schedules queries based on how much new data they're actually yielding. It's resource-capped end to end because an earlier version wasn't: the merge step once ballooned past 10GB of RAM and the kernel OOM-killer stalled the entire host. Every cap below (systemd memory limits, per-container Docker caps, DuckDB spill-to-disk) exists because of that incident.

flowchart TB
    subgraph systemd["systemd unit — MemoryHigh=4G · MemoryMax=6G · Nice=10"]
        LOOP[collector loop]
    end
    LOOP -->|spawns| D1["scraper container\n(docker --memory 4g)"]
    LOOP -->|spawns| D2["scraper container\n(docker --memory 4g)"]
    D1 --> RAW[(raw batches)]
    D2 --> RAW
    RAW --> MERGE["DuckDB incremental merge\nmemory_limit=3GB → spills to disk"]
    MERGE --> DATA[(deduped dataset)]

Read the full war story: Fixing OOM crashes in a 24/7 scraper

🎯 What This System Does

Stage	Tool	Output
Collect	Docker scraper engine, batch scheduler	2.2M+ reviews from 17,900+ places
Analyze	XLM-RoBERTa sentiment, LDA topics, KeyBERT keywords	Sentiment scores, 9 aspect categories, topic clusters
Visualize	8-page animated dashboard with maps, radar, time-series	Rankings, maps, trends, word clouds, radar plots, comparison matrices
Export	CSV/Parquet/JSON + HTML Executive Report	Excel-ready files + standalone HTML report
Loop	24/7 continuous collector with auto-resume	Fresh data every N hours, forever

📊 Real Data — Collected & Growing (as of 2 July 2026)

 2,778 places  ·  480,201 reviews  —  Petaling Jaya
 1,588 places  ·  255,199 reviews  —  Subang Jaya
 2,254 places  ·  202,642 reviews  —  Kajang
 1,780 places  ·  201,417 reviews  —  Cheras
 1,164 places  ·  189,989 reviews  —  Putrajaya
 1,546 places  ·  167,528 reviews  —  Cyberjaya
   991 places  ·  152,989 reviews  —  Bangi
   694 places  ·  145,813 reviews  —  Bangsar
   988 places  ·   53,990 reviews  —  Puchong
   849 places  ·   51,302 reviews  —  Seri Kembangan
   380 places  ·   22,322 reviews  —  Dengkil
   376 places  ·   15,733 reviews  —  Sepang
   551 places  ·   11,616 reviews  —  Nilai
   173 places  ·    5,072 reviews  —  Salak Tinggi
 + other areas (Serdang, unclassified suburbs, …)
────────       ───────────
17,914 places · 2,241,989 unique reviews · 1,267 merged batches

Still growing, unattended. The collector runs 24/7 as a systemd service with a yield-aware scheduler: each of the 470 active queries is re-scheduled based on how many new reviews its last run actually produced — a query yielding 3,000 fresh reviews comes back in hours, while a saturated one cools down for weeks. Growth from 455K reviews (June 2026) to 2.24M (July 2026) happened without manual intervention.

🚀 Quick Start

git clone https://github.com/ahmadafif5321/GoogleMapScrapper.git
cd GoogleMapScrapper
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

The One-Liner

python main.py full "clinic in Cyberjaya"

Scrapes reviews from Google Maps, runs sentiment analysis, generates insights. Ready for dashboard.

🧰 Command Arsenal

Command	What it does	Example
`scrape`	Pull reviews from Google Maps	`python main.py scrape "clinic in KL" "klinik in PJ"`
`analyze`	Sentiment + topics + insights	`python main.py analyze reviews.parquet -s vader`
`full`	Scrape then analyze in one shot	`python main.py full "cafe in Bandung"`
`collect`	Batch collector — hours to days	`python main.py collect -q queries.txt -b 3 -d 120`
`dashboard`	Launch 6-page Streamlit UI	`python main.py dashboard -p 8080`
`export`	Dump everything as CSV	`python main.py export`
`generate-queries`	Auto-spawn 332 search terms	`python main.py generate-queries`
`stats`	Live collection progress	`python main.py stats`

`collect` — The Heavy Lifter

# One-shot: work through all queries once
python main.py collect -q scraper/queries_full.txt

# 24/7 mode: re-scrape every 6 hours indefinitely
python main.py collect --continuous --cycle-hours 6

# Conservative (safe, slower)
python main.py collect -b 2 -d 180

# Aggressive (fast, riskier)
python main.py collect -b 5 -d 60

# Resume from crash: just run the same command
python main.py collect -q scraper/queries_full.txt --continuous

# Start fresh
python main.py collect --reset

Flag	Default	Description
`-q`	`scraper/queries.txt`	Query file (one per line)
`-b`	`2`	Batch size — queries per Docker run
`-d`	`180`	Cooldown between batches (seconds)
`-m`	`0` (unlimited)	Stop after N batches
`--continuous`	off	Re-run all queries every cycle
`--cycle-hours`	`24`	Hours between cycles
`--reset`	off	Wipe progress, start fresh

`analyze` — Two Sentiment Engines

# VADER — instant, no GPU, English/Malay
python main.py analyze reviews.parquet -s vader

# XLM-RoBERTa Transformer — multilingual, higher accuracy
python main.py analyze reviews.parquet -s transformer

`generate-queries` — Maximum Coverage

Generates 332 queries across 8+ cities × 25 medical categories (BM + EN), or 500+ queries for broader coverage. Pre-built files included:

File	Queries	Target
`scraper/queries_100places.txt`	50	~100-200 places, quick run
`scraper/queries_full.txt`	332	~500-1,000+ places, comprehensive
`scraper/queries_expansion.txt`	80+	Additional cities, extra coverage
`scraper/queries_ondemand.txt`	Custom	Specific businesses / Google share links

📈 The Dashboard — 8 Pages with Animations

Launch with python main.py dashboard. Opens at http://localhost:8501.

Page	What You See
📊 Executive Summary	Animated KPI cards, top 3 insights, health gauges, 1-click HTML report export
📈 Overview & Rankings	Place leaderboard with gradient tables, rating histogram, top/bottom performers
😊 Sentiment Analysis	Donut charts, stacked sentiment-by-rating, per-place breakdowns, sentiment health score
🔍 Aspect Analysis	9-dimension aspect breakdown (service, price, quality, ambiance, location, etc.)
🔑 Keywords & Topics	Word cloud, n-gram phrases, LDA topic clusters, keyword-sentiment correlation
⚔️ Competitor Comparison	Side-by-side benchmarks, radar charts, aspect heatmap matrix, competitive advantages
💡 Insights & Roadmap	Strengths, complaint categories, prioritized improvement roadmap, CSV/HTML export
🗺️ Map View	Interactive geographic map with sentiment overlay, place clustering, location rankings

✨ What's New in v2.0

Feature	Description
CSS Animations	Fade-in, slide-in, pulse, glow, shimmer, counter animations across all pages
Animated KPI Cards	Gradient cards with auto-animated counters, trend arrows, hover effects
Global Filter Bar	Filter by rating range, sentiment, category, and place — affects all charts
Map View	Plotly scatter mapbox with sentiment coloring, review count sizing, zoom/pan
Radar Charts	Multi-dimensional spider chart for competitor aspect comparison
Sentiment Health Score	Single metric: positive% − negative% with gauge visualization
HTML Report Generator	One-click standalone executive report with KPI tables and recommendations
Donut & Gauge Charts	Modern chart types replacing basic pie/bar where appropriate
Professional Theme	Custom CSS: gradient hero banners, scrollbar styling, card shadows, tooltips

🔄 The Collector Engine

┌─ CYCLE ─────────────────────────────────────────────────┐
│                                                          │
│  ┌─ BATCH LOOP ──────────────────────────────────────┐  │
│  │                                                    │  │
│  │  1. Pull N queries from queue                      │  │
│  │  2. Spin up Docker scraper    ┌──────────────────┐ │  │
│  │  3. Google Maps → JSON        │  RESUME-ABLE     │ │  │
│  │  4. Parse → parquet           │  Every batch      │ │  │
│  │  5. Save state to JSON        │  saved to disk    │ │  │
│  │  6. Cooldown (120s default)   └──────────────────┘ │  │
│  │  7. Repeat until queue empty                       │  │
│  │                                                    │  │
│  │  Every 10 batches: MERGE + EXPORT CSV              │  │
│  └────────────────────────────────────────────────────┘  │
│                                                          │
│  If --continuous: wait N hours → reset queue → repeat    │
└──────────────────────────────────────────────────────────┘

Crash-proof: progress saved after every single batch. Kill the process, reboot, Ctrl+C — re-run the exact same command and it picks up where it left off.

State file: scraper/collector_state.json — tracks every query status, timestamps, and counts.

📁 Architecture

GoogleMapScrapper/
│
├── main.py                ╸ CLI hub — 9 commands, argparse-driven
├── config.py              ╸ Settings: scraper limits, aspect cats, NLP models
│
├── scraper/
│   ├── run_scraper.py     ╸ Docker interface to gosom/google-maps-scraper
│   ├── collector.py       ╸ Batch scheduler, merge engine, CSV exporter
│   ├── query_generator.py ╸ Auto-generates 332+ queries across cities
│   ├── queries_*.txt      ╸ Pre-built query files (100 / full / expansion / ondemand)
│   └── collector_state.json ╸ Runtime: progress + yield-aware schedule (17.9K+ places, 2.2M+ reviews)
│
├── analytics/
│   ├── preprocess.py      ╸ Text cleaning, tokenization, filtering
│   ├── sentiment.py       ╸ VADER + XLM-RoBERTa, aspect-based scoring
│   ├── topics.py          ╸ TF-IDF, KeyBERT, LDA, n-gram extraction
│   ├── insights.py        ╸ Negative review mining, strengths, recommendations
│   ├── compare.py         ╸ Cross-entity competitor benchmarking
│   └── pipeline.py        ╸ Full orchestrator: raw → insights in one call
│
├── dashboard/
│   ├── app.py             ╸ Streamlit 8-page interactive web UI (v2.0)
│   ├── components.py      ╸ Reusable animated components (KPI cards, filters, reports)
│   └── styles.py          ╸ Custom CSS (animations, gradients, card themes)
│
├── data/
│   ├── raw/               ╸ Docker scraper JSON output (25GB+)
│   └── processed/         ╸ Parquet + CSV + JSON analytics (13GB+)
│
├── docker-compose.yml     ╸ Scraper container definition
└── requirements.txt       ╸ Python deps

⚙️ Configuration — `config.py`

Scraper Tuning

SCRAPER_CONCURRENCY = 4        # Parallel browser tabs
SCRAPER_DEPTH = 15             # Scroll depth per search
SCRAPER_EXTRA_REVIEWS = True   # ~300 reviews per place
SCRAPER_TIMEOUT = 7200         # 2h max per run

# Geo-targeting (optional — set to activate)
SCRAPER_GEO = None             # "lat,lon"
SCRAPER_RADIUS = None          # meters
SCRAPER_GRID_BBOX = None       # "minLat,minLon,maxLat,maxLon"

NLP Pipeline

SENTIMENT_MODEL = "cardiffnlp/twitter-xlm-roberta-base-sentiment"
EMBEDDING_MODEL = "paraphrase-multilingual-MiniLM-L12-v2"
MAX_REVIEWS_PER_PLACE = 300

Aspect Categories — 9 Dimensions

service · price · quality · ambiance · location · wait_time · menu_variety · portion · delivery

Each with bilingual keywords (EN + BM/ID). Extensible: add your own in config.py.

🔧 Usage Recipes

Scrape clinics across 10 cities

python main.py collect -q scraper/queries_full.txt -b 3 -d 120
python main.py analyze data/processed/collected_all_reviews.parquet -s vader
python main.py dashboard

Run 24/7 collector in background

nohup python main.py collect -q scraper/queries_full.txt --continuous --cycle-hours 6 &
python main.py stats      # check progress

Export everything for Excel / Power BI

python main.py export
# → data/processed/all_reviews.csv      (all reviews)
# → data/processed/places_summary.csv    (per-place stats)
# → data/processed/sentiment_by_place.csv
# → data/processed/recommendations.csv
# → data/processed/aspect_summary.csv
# → data/processed/comparison_ranking.csv

Target specific competitors

echo "Klinik Contoh Cyberjaya" > targets.txt
echo "Hospital Contoh" >> targets.txt
python main.py full "Klinik Contoh Cyberjaya" "Hospital Contoh" "Klinik Contoh 2 Sepang"

Use Google Maps share links directly

The scraper now accepts Google Maps share URLs (e.g., https://share.google/<share-id>) as queries — drop them in scraper/queries_ondemand.txt and run.

🖥️ Scale Guide

Scale	Places	Reviews	CPU	RAM	Time
Small	<30	<10K	2 cores	4GB	15 min
Medium	30-200	10K-60K	4 cores	8GB	1-3 hrs
Large	200-1,000	60K-300K	8 cores	16GB	6-24 hrs
City-wide	1,000+	300K+	16 cores	32GB	Days

Sentiment: VADER runs in seconds (even on 100K+ reviews). Transformer needs GPU for large datasets.

🐛 Troubleshooting

Problem	Fix
Docker not running	`sudo systemctl start docker`
Only 8 reviews per place	Re-scrape with latest code (bug fixed)
Collector crashed	Re-run same command — auto-resumes
Google rate limiting	Increase delay: `-d 300`, reduce batch: `-b 2`
Memory errors	Close other apps, reduce `-b` to 1
`ModuleNotFoundError`	`pip install sentencepiece tiktoken` (transformer mode)
Port 8501 taken	`python main.py dashboard -p 8080`
Empty results	Query might be too niche — broaden it

📄 License

MIT — see LICENSE

🙏 Credits

gosom/google-maps-scraper — the Docker scraper engine
HuggingFace Transformers — XLM-RoBERTa multilingual sentiment
Streamlit — dashboard framework
KeyBERT — keyword extraction

Name		Name	Last commit message	Last commit date
analytics		analytics
dashboard		dashboard
scraper		scraper
.gitignore		.gitignore
COLLECTOR_OPS.md		COLLECTOR_OPS.md
README.md		README.md
SCRAPING_GUIDE.md		SCRAPING_GUIDE.md
config.py		config.py
docker-compose.yml		docker-compose.yml
main.py		main.py
megu-collector.service		megu-collector.service
requirements.txt		requirements.txt
run_continuous_collector.sh		run_continuous_collector.sh

Navigation Menu

Repository files navigation

🗺️ Google Maps Review Intelligence Platform

🏗️ Architecture — 24/7 Production Collector

🎯 What This System Does

📊 Real Data — Collected & Growing (as of 2 July 2026)

🚀 Quick Start

The One-Liner

🧰 Command Arsenal

`collect` — The Heavy Lifter

`analyze` — Two Sentiment Engines

`generate-queries` — Maximum Coverage

📈 The Dashboard — 8 Pages with Animations

✨ What's New in v2.0

🔄 The Collector Engine

📁 Architecture

⚙️ Configuration — `config.py`

Scraper Tuning

NLP Pipeline

Aspect Categories — 9 Dimensions

🔧 Usage Recipes

Scrape clinics across 10 cities

Run 24/7 collector in background

Export everything for Excel / Power BI

Target specific competitors

Use Google Maps share links directly

🖥️ Scale Guide

🐛 Troubleshooting

📄 License

🙏 Credits

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

🗺️ Google Maps Review Intelligence Platform

🏗️ Architecture — 24/7 Production Collector

🎯 What This System Does

📊 Real Data — Collected & Growing (as of 2 July 2026)

🚀 Quick Start

The One-Liner

🧰 Command Arsenal

collect — The Heavy Lifter

analyze — Two Sentiment Engines

generate-queries — Maximum Coverage

📈 The Dashboard — 8 Pages with Animations

✨ What's New in v2.0

🔄 The Collector Engine

📁 Architecture

⚙️ Configuration — config.py

Scraper Tuning

NLP Pipeline

Aspect Categories — 9 Dimensions

🔧 Usage Recipes

Scrape clinics across 10 cities

Run 24/7 collector in background

Export everything for Excel / Power BI

Target specific competitors

Use Google Maps share links directly

🖥️ Scale Guide

🐛 Troubleshooting

📄 License

🙏 Credits

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`collect` — The Heavy Lifter

`analyze` — Two Sentiment Engines

`generate-queries` — Maximum Coverage

⚙️ Configuration — `config.py`