DropRead

DropRead is an autonomous AI agent pipeline that researches, synthesizes, and schedules weekly newsletters on any highly specific topic. Subscribe to any topic and it lands in your inbox on a schedule.

This project is written in Python and FastAPI, uses LangGraph for the agent pipeline, Groq (GPT-OSS 120B) for synthesis, FlashRank for lightweight reranking, pgvector for semantic caching, NewsAPI + crawl4ai for live web research, and Langfuse for LLM call tracing.

Live Website: https://dropread.site/

---

How it Works

Every topic first passes through a Gatekeeper — a structured-output LLM call that validates, canonicalizes, and generates search queries — before ever reaching the pipeline. The LangGraph pipeline itself has 5 nodes with PostgreSQL checkpointing for resumable execution, and every run is traced end-to-end in Langfuse (tagged by pipeline_mode and keyed to the digest ID):

gatekeeper → search → web_crawl → validation → rerank → synthesis

0. Gatekeeper — validates the topic is a legitimate, safe, newsletter-worthy subject (not gibberish, unsafe, unanswerable, or a one-off factual lookup); canonicalizes it (e.g. "AMD" → "Advanced Micro Devices") for consistent caching; generates a broad + narrow query pair
1. Search — queries NewsAPI with the broad query, automatically retrying with the narrower fallback query if the first returns zero results
2. Web crawl — tiered fetching: curl_cffi (TLS fingerprint spoofing) → crawl4ai (Playwright/Chromium for JS-heavy sites) → search snippet fallback. WAF detection skips challenge pages automatically.
3. Validation — LLM filters crawled pages to only those genuinely relevant to the topic
4. Rerank — FlashRank (nano model, ~4MB) scores each page against the topic; top-N kept
5. Synthesis — GPT-OSS 120B (via Groq) synthesizes a readable newsletter from the surviving sources, using a mode-specific prompt (news_roundup / deep_dive)

• Semantic cache: two-tier lookup — an exact match on a normalized cache slug first (instant), falling back to pgvector HNSW cosine similarity on BAAI/bge-small-en-v1.5 embeddings (≥ 0.82 = hit) if no exact match exists. Cached results older than 3 days get a lightweight NewsAPI freshness check before being trusted, rather than being served blindly.
• Scheduler: APScheduler fires every 15 minutes inside FastAPI's lifespan. next_delivery is bumped before pipeline execution — a failed run never causes infinite retries.
• Observability: every pipeline run is traced in Langfuse via a CallbackHandler bound to the digest ID, with each LLM call (gatekeeper, validation, synthesis) visible as a step in the trace — makes it possible to inspect exactly what a model saw and returned for any given digest, not just the final output.

---

Stack

Layer	Tech
Backend	FastAPI + async SQLAlchemy + PostgreSQL (Supabase)
AI pipeline	LangGraph + LangChain + Groq (GPT-OSS 120B for synthesis, GPT-OSS 20B for the gatekeeper)
ML models	BAAI/bge-small-en-v1.5 (embeddings), FlashRank nano (reranking)
Vector search	pgvector with HNSW index
Web crawling	curl_cffi + crawl4ai (Playwright/Chromium) + trafilatura
Search	NewsAPI
Frontend	HTMX + AlpineJs + Jinja2
Email	Resend
Deployment	AWS EC2 (t3.small) + Docker + Caddy (auto HTTPS)
CI/CD	GitHub Actions
Observability	CloudWatch (infra logs) + Langfuse (LLM call tracing)

Features

1. Input gatekeeper - LLM-validated topic gate rejects gibberish, unsafe content, prompt injection attempts, and non-newsletter-worthy topics before any search/crawl cost is spent
2. Live pipeline status - HTMX polls every 2s; triggers a digest-done event when ready
3. Email subscriptions - per-digest schedule (day + time), stored in UTC, displayed in local timezone
4. Digest archive - all past emailed digests preserved and browsable per topic
5. Two-tier semantic cache - exact-slug and embedding-similarity lookups return cached results instantly, no LLM call, with freshness triage on aged entries
6. LLM tracing - every pipeline run traced in Langfuse, tagged by pipeline mode, for step-by-step debugging of what each model call actually saw and returned
7. Security - double-submit cookie CSRF, SSRF protection, SlowAPI rate limiting, DOMPurify XSS sanitization
8. Kill switch - synthesis node checks if digest was deleted mid-run before spending tokens

---

Installation & Setup

1. Configure Environment Variables

Create .env.local in root directory with following variables:

POSTGRES_USER=
POSTGRES_PASSWORD=
POSTGRES_HOST=
POSTGRES_PORT=
POSTGRES_DB=
SECRET_KEY=
GROQ_API_KEY=
NEWSAPI_KEY=
RESEND_API_KEY=
GOOGLE_CLIENT_ID=
GOOGLE_CLIENT_SECRET=
RESEND_FROM_EMAIL=onboarding@resend.dev
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_BASE_URL=https://cloud.langfuse.com
APP_ENV=local
APP_URL=http://127.0.0.1:8000
ALLOWED_ORIGINS=*

For SECRET_KEY, run:

python -c "import secrets; print(secrets.token_hex(32))"

2. Prepare the Database

• Option 1 - Supabase (recommended) - it comes with pgvector enabled

  Create a free project at supabase.com. Grab connection details from Settings → Database and fill in the POSTGRES_* vars.

• Option 2 — Local PostgreSQL (15+ with pgvector installed):

  Install pgvector first:

  # Mac
  brew install pgvector
  # Ubuntu
  apt install postgresql-15-pgvector

Then create the database:

  psql -U postgres -c "CREATE DATABASE dropread;"

Set POSTGRES_HOST=localhost and POSTGRES_DB=dropread in .env.local.

Note: Supabase recommended for local dev too, free tier is sufficient. Migrations in step 5 handle the vector extension and HNSW index automatically.

3. Setup Google OAuth

1. Go to console.cloud.google.com
2. Create a project → APIs & Services → Credentials → Create OAuth 2.0 Client ID
3. Add http://127.0.0.1:8000/auth/google/callback to Authorized redirect URIs
4. Copy client ID and secret into .env.local

4. Grab other API keys

• Groq: console.groq.com
• NewsAPI: newsapi.org
• Resend: resend.com
• Langfuse: cloud.langfuse.com (free tier is sufficient)

Note: for local dev, Resend allows sending from onboarding@resend.dev to your registered email only. Set RESEND_FROM_EMAIL=onboarding@resend.dev in .env.local. For production, use your verified domain.

5. Install dependencies and run

uv sync
uv run playwright install chromium
uv run alembic upgrade head
uv run uvicorn src.ai_newsletter.app:app --reload

Note: The first time you generate a topic, it will take 1-2 minutes to download ML models.

Linux users should run uv run playwright install-deps chromium

Open http://127.0.0.1:8000

6. Running tests

uv run pytest tests/ -v

20 tests across 4 classes: delivery time logic, semantic cache (mocked DB, threshold 0.82), pipeline route auth + validation, and email dispatch.

7. CI/CD

build-base → test → migrate → deploy

• build-base skips if pyproject.toml / uv.lock unchanged (hash-based)
• migrate runs alembic upgrade head against production Supabase before deploy
• deploy SSH into EC2, pulls base image from GHCR, rebuilds app layer only

Note: Base image includes Chromium, PyTorch CPU, and the embedding + reranking models pre-downloaded. App layer rebuilds in seconds on code-only changes.