Blog Knowledge Base & Semantic Search Engine

An automated system to scrape, summarize, and semantically index blog articles into a personal knowledge base. It extracts key insights using Large Language Models (LLMs), stores them as local Markdown files, syncs them to Notion, and provides a vector-based search interface.

🚀 Features

• Deep Scraping: Extracts clean text and metadata from any blog URL using Trafilatura and BeautifulSoup4.
• AI Summarization: Generates highly detailed, structured summaries using Google Gemini (with multiple model fallbacks) or locally via Ollama (qwen3:8b).
• Keyword Extraction: Automatically identifies key topics for better organization.
• Vector Search: Uses FAISS and Sentence-Transformers for lightning-fast semantic search across your entire archive.
• Notion Integration: Automatically syncs processed articles, summaries, and metadata to a Notion database.
• Local Archive: Maintains a local directory of Markdown files for offline access and long-term storage.

🛠️ Architecture

1. Input: Feed a URL to the system.
2. Processing:
   • Scrape content and metadata.
   • Extract keywords.
   • Generate an exhaustive summary using LLMs.
3. Storage:
   • Save as a structured Markdown file in knowledge_base/.
   • Index the summary and title into a FAISS vector database.
   • Push metadata and summary to a connected Notion page.
4. Retrieval: Query the index via semantic search to find relevant articles even without exact keyword matches.

📋 Prerequisites

• Python 3.8+
• Ollama (optional, for local LLM fallback)
• Gemini API Key (for high-quality summarization)
• Notion Integration Token (for Notion sync)

⚙️ Setup

1. Clone the repository and navigate to the project directory.
2. Create a virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

4. Configure Environment Variables: Create a .env file in the root directory with the following:

   GEMINI_API_KEY=your_gemini_api_key
   NOTION_TOKEN=your_notion_integration_token
   NOTION_DATABASE_ID=your_notion_database_id

📖 Usage

Processing a New Article

Run the batch script (on Windows) or the Python script directly:

# Using the batch script
.\run_blog.bat https://example.com/blog-post

# Using Python
python main.py https://example.com/blog-post

Searching Your Knowledge Base

Perform a semantic search across your indexed articles:

python search.py "your search query here" --top_k 5

📚 Detailed Documentation

For more in-depth information about individual modules, refer to the following documentation files:

• Main Application (main.md)
• Scraper (scraper.md)
• Summarizer (summarizer.md)
• Embeddings & Vectorization (embeddings.md)
• Keyword Extraction (keywords.md)
• Search Functionality (search.md)

🧪 Technologies Used

• LLMs: Google Gemini (Pro/Flash), Ollama (Local)
• Vectors: FAISS (Facebook AI Similarity Search)
• Embeddings: HuggingFace all-MiniLM-L6-v2
• Scraping: Trafilatura, BeautifulSoup
• Productivity: Notion API