Sunnah.com Scraper

A Python scraper designed to extract hadith collections, books, and individual narrations from Sunnah.com. Featuring an interactive CLI, resume capability, and rich metadata extraction.

✨ Features

• Interactive Menu: User-friendly CLI for guided scraping or specific targeting.
• Hierarchical Output: Organized data structure (data/collections/{slug}/books/*.json).
• Resume Capability: Automatically tracks progress and skips already scraped books.
• Rich Metadata: Captures collection descriptions, "About" sections, and supported languages.
• Detailed Hadiths: Bilingual text (Arabic/English), full grading (e.g., Sahih, Hasan), and various reference systems.
• Bilingual Support: Preserves both English and Arabic titles and content.
• Resilient: Robust network manager with retries, backoff, and concurrent fetching for translations.

🚀 Installation

1. Clone the repository:

   git clone <repository-url>
   cd sunnah-com-scraper

2. Install dependencies:

   pip3 install -r requirements.txt

🛠 Usage

The scraper supports both an interactive menu and direct CLI flags.

1. Interactive Menu (Recommended)

Launch the interactive explorer to select collections and books manually:

python3 scraper.py --menu

2. CLI Options

For automation or quick scrapes:

python3 scraper.py [OPTIONS]

Option	Description
--menu	Launch the interactive console menu (default if no args).
--slug SLUG	Target a specific collection (e.g., bukhari, muslim).
--books N	Limit the number of books to scrape per collection.
--langs L1,L2	Filter for specific languages (e.g., english,arabic).
--workers N	Set number of concurrent workers for translation fetching (default: 5).

📁 Data Structure

Data is stored in a clean, hierarchical format:

data/
└── collections/
    └── {collection_slug}/
        ├── metadata.json       # Collection info, description, "About"
        └── books/
            ├── 1_revelation.json
            ├── 2_belief.json
            └── ...

Metadata Example (metadata.json)

Contains high-level details about the collection:

• name: Full title in English and Arabic.
• description: Brief introduction from the main page.
• about: Detailed historical context/author bio.
• supported_languages: List of available translations.

Hadith Item Example

Each hadith preserves its context and grading:

{
  "hadith_id": "h100080",
  "translations": {
    "english": "...",
    "arabic": "..."
  },
  "grading": [
    { "grade": "Sahih", "by": "Al-Bukhari" }
  ],
  "references": {
    "Reference": "Sahih al-Bukhari 8",
    "In-book reference": "Book 2, Hadith 1"
  }
}

🔄 State Management

The scraper uses scraper_state.json to keep track of completed books.

• If interrupted, simply rerun the script to resume where it left off.
• Use Option 3 in the menu to Reset State if you wish to start fresh.

📝 Common Slugs

Collection	Slug
Sahih al-Bukhari	bukhari
Sahih Muslim	muslim
Sunan an-Nasa'i	nasai
Sunan Abi Dawud	abudawud
Jami' at-Tirmidhi	tirmidhi
Sunan Ibn Majah	ibnmajah
Muwatta Malik	malik
40 Hadith Nawawi	nawawi40
Riyad as-Salihin	riyadussalihin

---

Note: This scraper is intended for educational purposes and personal research. Please respect Sunnah.com's terms of service and avoid aggressive scraping speeds.