Grivredr - AI-Powered Web Scraper Generator

Train once, automate forever. Grivredr uses Claude AI to learn how to navigate government portals, then generates production-ready Python scrapers that work without ongoing AI costs.

🎯 Learn any portal in 2-3 minutes • 💰 ~$0.12 one-time cost • 🚀 Unlimited free usage after

🎬 Quick Start

# 1. Clone and setup
git clone https://github.com/yourusername/grivredr.git
cd grivredr
./quickstart.sh

# 2. Train your first portal
python cli/train_cli.py abua_sathi --district ranchi

# 3. Test it
python tests/test_abua_sathi_live.py

📖 New here? Check out GETTING_STARTED.md for a complete walkthrough.

✨ What Makes Grivredr Special?

🤖 AI-Powered Discovery

Claude Vision analyzes form structure from screenshots
Interactive exploration automatically clicks dropdowns and detects cascading fields
Hybrid strategy: Fast Playwright + intelligent Browser Use AI fallback
JavaScript monitoring captures AJAX calls and dynamic behavior

🔧 Production-Ready Code Generation

Self-healing: Validates and fixes generated code automatically (3 attempts)
Pattern library: Learns from successful scrapers to improve future ones
Smart templates: Handles Select2, cascading dropdowns, AJAX submissions
Zero AI costs after training - scrapers run standalone

🎯 Smart Features

Network tab analysis: Detects APIs and generates direct HTTP calls (5-10x faster than browser)
Confidence scoring: Only proceeds when form understanding is >70%
Human fallback: Record your actions if AI fails (becomes ground truth)
Cost optimization: AI response caching, model selection per task

💰 Cost Model

Phase	Cost	Frequency
Training	~$0.12 per portal	One-time only
Execution	$0.00	Unlimited forever

Example: Train 10 portals ($1.20) → Submit unlimited requests ($0.00)

🏗️ How It Works

Grivredr uses a 4-phase AI agent pipeline:

┌─────────────────────────────────────────────────────────────────┐
│ Phase 1: Form Discovery (30-60s)                                │
│ • Claude Vision analyzes screenshots                             │
│ • Interactive exploration (dropdowns, cascading fields)          │
│ • Hybrid: Playwright first, Browser Use AI if needed            │
│ • Confidence score: >0.6 to proceed                             │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ Phase 2: JavaScript Analysis (10-20s)                           │
│ • Monitors JS runtime during form interaction                    │
│ • Detects AJAX calls, dynamic behavior, event handlers          │
│ • Identifies API endpoints for direct HTTP calls                │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ Phase 3: Test Validation (30-60s)                               │
│ • Tests empty submission (finds required fields)                │
│ • Tests field types, cascading dropdowns                        │
│ • Full submission with mock data                                │
│ • Confidence score: >0.7 to proceed                             │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ Phase 4: Code Generation (20-40s)                               │
│ • Claude Opus generates Python scraper                          │
│ • Self-healing validation loop (3 attempts)                     │
│ • Stores pattern in knowledge base                              │
│ • Saves to outputs/generated_scrapers/                          │
└─────────────────────────────────────────────────────────────────┘

Total: ~2-3 minutes, ~$0.12 per portal

📦 Installation

Automated Setup (Recommended)

./quickstart.sh

Handles everything: dependencies, Playwright, and configuration.

Manual Setup

1. Install Python Dependencies

pip install -r requirements.txt

2. Install Playwright Browsers

python -m playwright install chromium

3. Configure API Key

cp .env.example .env
# Edit .env and add your MegaLLM API key

Get your API key: https://app.mega-llm.com

🚀 Usage

Train a New Portal

# Basic training
python cli/train_cli.py <portal_name> --district <district>

# Example: Train Jharkhand's Abua Sathi portal
python cli/train_cli.py abua_sathi --district ranchi

# With custom URL
python cli/train_cli.py new_portal --district mumbai \
  --url https://portal.example.com/complaint

Training Options:

--headless                # Run browser in headless mode
--no-hybrid               # Disable hybrid discovery (Playwright only)
--browser-use-first       # Try Browser Use AI first
--no-recording            # Disable human recording fallback

Use Generated Scraper

from outputs.generated_scrapers.ranchi_district.portals.abua_sathi import AbuaSathiScraper

async def submit_complaint():
    scraper = AbuaSathiScraper(headless=True)

    result = await scraper.submit_grievance({
        'name': 'John Doe',
        'contact': '9876543210',
        'village_name': 'Test Village',
        'description': 'Street light not working'
    })

    print(f"Success: {result['success']}")
    if result.get('tracking_id'):
        print(f"Tracking ID: {result['tracking_id']}")

Test Generated Scraper

# Test live scraper (visible browser)
python tests/test_abua_sathi_live.py

# Run all tests
pytest tests/

🎯 Features

Core Discovery

✅ Hybrid Discovery Strategy - Intelligently combines Playwright + Browser Use AI
✅ Claude Vision - Analyzes form structure from screenshots
✅ Interactive Exploration - Automatically clicks dropdowns and detects fields
✅ Network Monitoring - Captures API calls and generates direct HTTP code
✅ Event Listener Detection - Inspects blur/focus/input handlers

Code Generation

✅ Self-Healing - Validates and fixes code automatically
✅ Pattern Library - Learns from successful scrapers
✅ API-Aware - Generates direct HTTP calls when possible (5-10x faster)
✅ Framework Detection - Handles Select2, Chosen.js, cascading dropdowns

Intelligence

✅ Native Anthropic SDK - Official Python SDK with MegaLLM
✅ LangChain Integration - Optional for advanced workflows
✅ AI Response Caching - Reduces costs on retries
✅ Multi-Agent System - Specialized agents for each phase

Supported Portal Types

✅ Simple HTML forms (POST)
✅ AJAX-based submissions
✅ Select2/Chosen.js dropdowns
✅ Cascading dropdowns (parent → child)
✅ Multi-step forms
✅ File uploads
✅ ASP.NET ViewState/EventValidation

📊 Example Portals

Grivredr has successfully trained on:

Jharkhand Portals

Abua Sathi - State grievance system with Select2 dropdowns
Ranchi Smart - City smart portal with category selection
Ranchi Municipal - Municipal complaint forms

Success Metrics

🎯 95%+ accuracy on form field detection
⚡ 2-3 minutes average training time
💰 $0.08-0.15 average cost per portal
✅ 100% success rate on generated scrapers

🐛 Debugging

Check Form Discovery Results

python scripts/check_discovery_results.py

Shows:

All form fields discovered
Dropdown detection (Select2, cascading)
Required field validation

View Training Session

cat data/training_sessions/portal_name_timestamp.json

Contains:

Form discovery results
Test validation results
Generated code
Cost breakdown
Confidence scores

Common Issues

Low confidence score during discovery

Try hybrid discovery (enabled by default)
Use --browser-use-first for complex forms

Generated scraper fails validation

Check session JSON for error details
Review screenshots in outputs/screenshots/
Human recording fallback will be offered

Cascading dropdown timeouts

Increase wait time in generated code
Check AJAX patterns in JS analysis results

📚 Documentation

Getting Started - Complete beginner's guide
Project Structure - Codebase organization
Architecture - System design details
Claude Code Guide - For Claude Code assistant
Contributing - Contribution guidelines
Status - Current features and roadmap

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Areas we'd love help with:

🌍 Testing on more government portals
🔧 Improving pattern detection
📝 Documentation and examples
🧪 Adding test coverage
🚀 Performance optimizations

🛠️ Tech Stack

AI: Anthropic Claude (Haiku, Sonnet, Opus) via MegaLLM
Browser Automation: Playwright + Browser Use
Language: Python 3.11+
Knowledge Base: SQLite + Optional ChromaDB
Testing: Pytest
Optional: LangChain for advanced workflows

⚠️ Known Limitations

CAPTCHA: Detected but requires human intervention
OTP: Requires real phone numbers (not automated)
Very slow AJAX (>10s): May timeout
reCAPTCHA: Detected but not bypassed

📜 License

MIT License - see LICENSE file for details.

🙏 Acknowledgments

Anthropic - For Claude AI models
MegaLLM - For affordable Claude API access
Playwright - For reliable browser automation
Browser Use - For AI-powered web interaction

📞 Support

Issues: GitHub Issues
Discussions: GitHub Discussions
Documentation: See docs/ directory

⚖️ Legal Notice

This tool is designed for legitimate civic engagement and automation. Users are responsible for:

Complying with website Terms of Service
Respecting rate limits and robot policies
Using scrapers ethically and legally

No warranty provided. Use at your own risk.

Built with ❤️ using Claude AI, Playwright, and Python

Star ⭐ this repo if Grivredr helps you automate government portals!

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
agents		agents
batch		batch
cli		cli
config		config
docs		docs
executor		executor
intelligence		intelligence
knowledge		knowledge
legacy		legacy
scraper_generator		scraper_generator
scripts		scripts
tests		tests
utils		utils
website_learner		website_learner
.env.example		.env.example
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
CONTRIBUTING.md		CONTRIBUTING.md
GETTING_STARTED.md		GETTING_STARTED.md
LICENSE		LICENSE
PROJECT_STRUCTURE.md		PROJECT_STRUCTURE.md
README.md		README.md
REPOSITORY_OVERVIEW.md		REPOSITORY_OVERVIEW.md
main.py		main.py
pyproject.toml		pyproject.toml
quickstart.sh		quickstart.sh
requirements.txt		requirements.txt
verify_setup.py		verify_setup.py

Latest commit

History

Folders and files

Repository files navigation