DevScraper - Advanced Job Scraper

A powerful, production-grade job scraper for remote development positions with focus on Nigeria-relevant opportunities.

Features

✨ Multi-Source Scraping

• Google Jobs (via SerpAPI)
• Twitter/X (via API v2)
• RemoteOK API
• LinkedIn (guest API)
• Jobicy & Remotive APIs (optional)

🚀 Performance & Architecture

• Concurrent execution: All scrapers run in parallel via ThreadPoolExecutor
• Retry logic: Exponential backoff for resilient API calls
• Deduplication: Removes redundant job listings
• Type hints: Full type annotations for code clarity
• Logging: Comprehensive logging to file and console

🔍 Smart Filtering

• Dev keyword filtering (React, Vue, TypeScript, etc.)
• Nigeria-relevance tagging
• Location-based sorting
• Configurable search queries

💾 Data Management

• JSON export (full job data)
• CSV export (easy spreadsheet import)
• Timestamped outputs to prevent overwrites

Setup

Prerequisites

• Python 3.10+
• pip or conda

Installation

# Clone repository
git clone https://github.com/Gbagamsyle/devscraper.git
cd devscraper

# Create virtual environment (optional but recommended)
python -m venv .venv
source .venv/bin/activate  # Linux/Mac
# or
.venv\Scripts\activate  # Windows

# Install dependencies
pip install -r requirements.txt

Configuration

Create a .env file in the project root:

# Required: Google Jobs (SerpAPI)
SERPER_KEY=your_serper_api_key

# Optional: Twitter/X Jobs
TWITTER_BEARER=your_twitter_bearer_token

Get API keys:

• SerpAPI: https://serper.dev/ (free tier available)
• Twitter: https://developer.twitter.com/en/portal/dashboard (requires v2 API access)

Usage

Run Complete Scrape

python job_runner.py

This will:

1. Run all enabled scrapers concurrently
2. Deduplicate jobs
3. Tag Nigeria-relevant positions
4. Export results to output/ directory

Output Files

Results are saved with timestamp:

• output/jobs_YYYYMMDD_HHMM.json - Complete job data
• output/jobs_YYYYMMDD_HHMM.csv - Spreadsheet-ready format
• logs/scraper.log - Detailed execution logs

Example Output

{
  "title": "Senior React Developer",
  "company": "Tech Startup",
  "location": "Remote",
  "posted": "2026-06-12",
  "salary": "$80,000 - $120,000",
  "apply_link": "https://...",
  "description": "We are hiring...",
  "source": "remoteok",
  "nigeria_relevant": true
}

Configuration

Edit config.py to customize:

# Search queries
NIGERIA_QUERIES = [...]
GLOBAL_QUERIES = [...]

# Scraper settings
TIMEOUT = 15  # Request timeout in seconds
RETRIES = 3   # Number of retry attempts
MAX_WORKERS = 4  # Concurrent API calls

# Toggle scrapers
CONFIG = {
    "sources": {
        "google": True,
        "twitter": True,
        "free_boards": True,
        "alternative": False,  # Jobicy & Remotive
    }
}

Project Structure

devscraper/
├── config.py              # Configuration & constants
├── utils.py               # Shared utilities, logging, retry logic
├── google_jobs.py         # Google Jobs scraper
├── twitter_jobs.py        # Twitter/X scraper
├── free_boards.py         # RemoteOK & LinkedIn scrapers
├── scrapers.py            # Alternative scrapers (Jobicy, Remotive)
├── job_runner.py          # Main orchestrator
├── requirements.txt       # Python dependencies
└── README.md              # This file

Architecture Improvements

Before → After

Aspect	Before	After
Code duplication	High (3 locations)	Eliminated (utils.py)
Error handling	Basic try/catch	Retry with exponential backoff
Execution	Sequential	Concurrent (3-4x faster)
Logging	Print statements	Structured logging (file + console)
Twitter	Stub (empty)	Full API v2 implementation
Type safety	None	Full type hints
Config	Hardcoded	Centralized in config.py
API secrets	Pushed to git ❌	Removed + .gitignore added ✓

Performance

On typical run with all scrapers:

• Google Jobs: ~50 results (1.5s)
• Twitter: ~0-50 results (requires API v2 access, ~5s)
• Free Boards: ~30-50 results (2s)
• Total time: ~8-10s (with concurrency) vs ~20s+ (sequential)

Memory: ~50-100MB (depends on job count)

Logging

Logs are written to logs/scraper.log:

2026-06-12 00:37:32 [job_scraper] [INFO] Starting job scraper...
2026-06-12 00:37:32 [job_scraper] [INFO] Running 3 scrapers concurrently...
2026-06-12 00:37:34 [job_scraper] [INFO] Google Jobs: 74 jobs
2026-06-12 00:37:34 [job_scraper] [INFO] Twitter/X: 0 jobs
2026-06-12 00:37:34 [job_scraper] [INFO] Free Boards: 27 jobs
2026-06-12 00:37:47 [job_scraper] [INFO] Complete: 101 jobs (50 Nigeria-relevant)

Error Handling

The system gracefully handles:

• ❌ Missing API keys (skips that scraper)
• ❌ Rate limiting (retries with backoff)
• ❌ Network timeouts (3 attempts)
• ❌ Invalid responses (logs & continues)

Development

Add New Scraper

1. Create your_scraper.py:

from config import TIMEOUT
from utils import retry_with_backoff, deduplicate_jobs, logger

@retry_with_backoff()
def fetch_jobs():
    # Your scraper logic
    pass

def run_your_scraper():
    return deduplicate_jobs(fetch_jobs())

2. Update job_runner.py:

from your_scraper import run_your_scraper

# In run_scrapers_concurrent():
scrapers.append(("Your Source", lambda: run_your_scraper()))

Troubleshooting

"Import 'X' could not be resolved"

• Install dependencies: pip install -r requirements.txt
• Use correct Python interpreter

"SERPER_KEY not set"

• Create .env file with your API key
• File must be in project root

"Twitter API 402 Payment Required"

• Twitter API v2 requires paid tier for search endpoint
• Disable in config.py if not available

Encoding errors on Windows

• Logging is now UTF-8 compatible
• Update to latest Python 3.11+

Future Enhancements

• Database persistence (SQLite/PostgreSQL)
• Salary range extraction & analysis
• Skill matching & job recommendations
• Web UI dashboard
• Email notifications
• Scheduled scraping (cron/APScheduler)
• GitHub Actions CI/CD

License

MIT

Contributing

Pull requests welcome! Please:

1. Fork repository
2. Create feature branch
3. Add tests
4. Submit PR

Support

Found a bug? Issues with scraping? Open an issue on GitHub or contact the maintainer.

---

Made with ❤️ for African developers