AI-powered annotation engine for academic papers — arXiv or your own PDF.
Annotagent reads, analyzes, and annotates research papers — generating inline highlights, contextual notes, and jargon definitions anchored directly to PDF coordinates. A built-in chat panel lets you interrogate any paper with full-text context, turning passive reading into active inquiry.
Live deployment · built by PitchFork
- Pick a source — paste an arXiv ID or drag-drop a PDF (≤25 MB). Uploads are signed straight to S3, then handed to the ingestion service.
- Metadata + extraction — the FastAPI service fetches the PDF, validates it, extracts text blocks and bounding boxes with PyMuPDF, and pulls title/abstract from the first page via a small OpenAI call. arXiv papers and user uploads share the same downstream pipeline.
- Annotation pipeline — text is split into paragraph-scale chunks and processed in parallel against a retrieval index. Each chunk produces structured annotations (
highlight,note,definition) with few-shot prompting, JSON repair, dedupe, optional cross-paper validation, deterministic text anchors, and bbox refinement against the real PDF layout. - Anchored overlays — annotations render as hand-drawn SVG underlines positioned with sub-page precision using text anchors resolved against the extracted page content, not fragile absolute coordinates.
- Paper-aware chat — ask questions with the full paper text injected as context. Responses stream with LaTeX math rendering. Transcripts are kept in MongoDB with a 24-hour TTL.
┌──────────────────────────────────────────────────────────────────┐
│ Next.js 15 App Router (Vercel) │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────────┐ │
│ │ PDF Workspace│ │ Chat Panel │ │ Library / Auth │ │
│ │ react-pdf + │ │ Streaming │ │ NextAuth v5 (JWT) │ │
│ │ SVG overlays │ │ AI SDK │ │ Credentials + Resend │ │
│ └──────┬───────┘ └──────┬───────┘ └────────┬───────────────┘ │
│ │ │ │ │
│ ┌──────┴─────────────────┴───────────────────┴───────────────┐ │
│ │ Server orchestration · lib/server-data.ts │ │
│ │ thin route handlers · S3 PDF cache · TTL chat · prefs │ │
│ └────────────┬─────────────────────────────────┬─────────────┘ │
└───────────────┼─────────────────────────────────┼────────────────┘
│ │
│ HMAC-signed │
│ browser → Python │
▼ ▼
┌──────────────────────────┐ ┌──────────────────────────┐
│ FastAPI Python service │ │ MongoDB Atlas │
│ arXiv + upload ingest │ │ papers · userpapers │
│ PyMuPDF extraction │ │ chats (TTL) · users │
│ LongRAG / Contextual │ │ ingestionevents (RL) │
│ retrieval + OpenAI │ └──────────────────────────┘
│ Pydantic validation │
└────────────┬─────────────┘ ┌──────────────────────────┐
│ │ AWS S3 │
└───────────────────▶│ arxiv/<id>.pdf │
│ user-uploads/<user>/.. │
└──────────────────────────┘
- Two paper sources — public arXiv IDs (deduplicated globally) or private user-uploaded PDFs (25 MB cap, scoped to the uploader).
- Annotation styles — choose between Default (technically literate non-specialist), Novice (defines all jargon, higher density), and Expert (focus on novelty, prior-work comparisons, lower density). Selected in the side panel before ingest; per-paper style is stored and pre-populated on reprocess.
- Annotation modes — saved per user. Fast uses LongRAG-style section-level units (no LLM calls during indexing, ~11% fewer annotations). Densest uses Contextual Retrieval (per-chunk situating blurbs + embeddings; denser annotations, slower indexing).
- Reprocess pathways — on a loaded paper, reprocess with Instant (skip LLM validation pass) or Thinking (full validation agent run). Style is editable per reprocess.
- Multi-stage annotation pipeline — few-shot generation, LLM-based JSON repair, dedupe, optional cross-page LLM validation, deterministic local cleanup, text-anchor resolution, and PDF-native bbox refinement using PyMuPDF text search.
- Rolling memory — the pipeline maintains compact in-process memory (paper state, defined terms, covered topics, recent annotations) across chunks so longer papers stay coherent without redundant annotations.
- Text-anchor resolution — annotations carry deterministic page-text offsets plus an occurrence index, so the same term on a page can be disambiguated and overlays stay aligned even when bboxes drift.
- Paper briefs — sampled early/middle/late chunks build a representative structural brief before annotation begins.
- Streaming chat with math — paper-bounded Q&A via the Vercel AI SDK, rendered with React Markdown + remark-math + KaTeX; chat history persists in MongoDB with a 24-hour TTL.
- PDF caching in S3 — arXiv PDFs are cached once and shared; uploads stay private under
user-uploads/<userId>/. The PDF route returns a short-lived presigned URL. - Per-user library — application-layer authorization through
userpapersjoins; arXiv papers are deduplicated, uploads are not. - Rate limiting — 10 ingestion authorizations per user per hour (sliding window, MongoDB-backed, configurable via
INGEST_HOURLY_LIMIT). - Direct browser→Python ingest — long-running jobs go straight to FastAPI using short-lived HMAC tokens minted by the authenticated Next.js route, so Vercel never has to hold a 5–10 minute connection.
| Layer | Technology |
|---|---|
| Frontend | Next.js 15 App Router, React 19, TypeScript, Tailwind CSS, Zustand, TanStack Query |
| PDF rendering | react-pdf with custom SVG annotation overlays |
| Chat | Vercel AI SDK, OpenAI streaming, KaTeX math rendering |
| Auth | NextAuth.js v5 (Credentials + JWT), bcryptjs, Resend for password-reset email |
| Database | MongoDB Atlas via Mongoose (+ official driver for NextAuth adapter) |
| Object storage | AWS S3 (presigned PUT for uploads, presigned GET for reads) |
| Ingestion service | FastAPI, PyMuPDF, LangChain text splitters, OpenAI |
| Retrieval | LongRAG (fast) / Contextual Retrieval with embeddings (dense), pluggable via python_service/providers/ |
| Deployment | Vercel (Next.js) · separately hosted Python service (Render/Fly/etc.) |
- Node.js 20.x
- Python 3.10+
- MongoDB Atlas cluster (or local
mongod) - AWS account with an S3 bucket
- Resend account (for password-reset email)
- OpenAI API key
git clone https://github.com/PitchFork-Innovation/annotagent.git
cd annotagent
npm install
python3 -m venv .venv
.venv/bin/pip install -r python_service/requirements.txtCopy .env.example to .env.local and fill in the values:
cp .env.example .env.local| Variable | Required | Description |
|---|---|---|
MONGODB_URI |
yes | MongoDB Atlas connection string |
NEXTAUTH_SECRET |
yes | 32+ byte random string — openssl rand -base64 32 |
NEXTAUTH_URL |
yes | Full URL of the deployment (e.g. http://localhost:3000) |
RESEND_API_KEY |
yes | Resend API key for password-reset email |
RESEND_FROM_EMAIL |
yes | Verified sender address in Resend |
AWS_REGION |
yes | Region of the S3 bucket |
AWS_ACCESS_KEY_ID |
yes | IAM credentials with bucket read/write |
AWS_SECRET_ACCESS_KEY |
yes | IAM credentials with bucket read/write |
S3_BUCKET |
yes | S3 bucket name |
PYTHON_SERVICE_URL |
yes | Base URL of the running FastAPI service |
PYTHON_SERVICE_SHARED_SECRET |
yes | HMAC secret shared with the Python service |
OPENAI_API_KEY |
yes | OpenAI API key (used by the Python service and chat) |
PYTHON_INGEST_TIMEOUT_MS |
no | Default 900000 (15 min) |
INGEST_HOURLY_LIMIT |
no | Per-user ingestion authorizations per hour (default 10) |
OPENAI_CHAT_MODEL |
no | Override default chat model |
OPENAI_ANNOTATION_MODELS |
no | Comma-separated override for annotation model |
OPENAI_ANNOTATION_TIMEOUT_SECONDS |
no | Per-request timeout for annotation calls |
KV_REDIS_URL / KV_REST_API_URL / KV_REST_API_TOKEN |
no | Used by the Python service only — shared progress storage on multi-instance deployments. Local dev falls back to temp-file storage |
In production, sensitive variables are required. Locally, missing or placeholder values fall back to safe defaults so npm run dev works even before MongoDB / S3 / Resend are wired up.
- MongoDB Atlas — create an M0 cluster, allowlist your IP (or
0.0.0.0/0for dev), setMONGODB_URI. The required collections (papers,userpapers,chats,passwordresettokens,ingestionevents, plus NextAuth'susers/accounts) and their indexes are created automatically by Mongoose. - AWS S3 — create a bucket, create an IAM user with
s3:PutObject,s3:GetObject,s3:DeleteObject,s3:ListBucketon that bucket. Configure CORS to allowPUTfrom the app origin. - Resend — get an API key, verify a sender domain or address.
- NextAuth —
openssl rand -base64 32and setNEXTAUTH_SECRET; setNEXTAUTH_URLto the deployment URL.
# Terminal 1 — Next.js
npm run dev
# Terminal 2 — Python ingestion service
npm run python:dev # alias for .venv/bin/python python_service/main.pyOpen http://localhost:3000, create an account, paste an arXiv ID or drop a PDF.
npm run lint
npm run typecheck
python3 -m py_compile python_service/main.pyCross-boundary changes (UI ↔ route ↔ server-data ↔ Python ↔ DB) should run all three.
app/ Next.js App Router routes
api/
auth/ NextAuth signup, password reset, [...nextauth]
chat/ Streaming paper Q&A
ingest/ arXiv ingest, authorize (HMAC), apply, progress
upload/ PDF upload init (presigned PUT), authorize
papers/[paperId]/ Workspace JSON, PDF proxy, reprocess flow
me/preferences User annotation-mode preference (GET/PATCH)
auth/ Sign-out form post target
paper/[paperId]/ Paper workspace page (server-loaded)
reset-password/ Password reset landing page
components/
workspace/ PDF viewer, annotation overlays, chat panel
landing-shell.tsx Home page (arXiv tab + upload tab, library grid)
auth-panel.tsx Sign in / up, password reset, style + mode picker
providers.tsx React Query + context providers
rich-text.tsx Markdown + LaTeX rendering
lib/
server-data.ts Cross-service orchestration (the only place that
touches MongoDB + S3 + Python in the same flow)
models/ Mongoose schemas (papers, userpapers, chats,
passwordresettokens, ingestionevents)
auth/ NextAuth v5 config + helpers
mongodb.ts Mongoose connection
mongodb-client.ts Official driver (NextAuth adapter + retry helper)
s3.ts Presigned URL + object helpers
python-service.ts Browser-side fetches to FastAPI
python-auth.ts HMAC token signing for browser → Python
chat-store.ts TTL-indexed chat read/write
rate-limit.ts Sliding-window per-user ingestion limiter
user-preferences.ts Annotation mode preference (fast | dense)
ingestion-schema.ts Zod validation for Python ingest payloads
types.ts Shared TypeScript contracts
env.ts Zod-validated env config
python_service/
main.py FastAPI entrypoint, prompts, pipeline
providers/ Retrieval providers (LongRAG, Contextual)
experiments/ Local experiments — not used in production
requirements.txt
docs/ Subsystem documentation (see AGENTS.md)
prds/ Product requirements documents
See AGENTS.md for the fast entry point. Subsystem detail lives in docs/:
docs/architecture.md— system map and end-to-end request flowsdocs/frontend.md— App Router pages and workspace UI invariantsdocs/api-server-data.md— route handlers andlib/server-data.tsdocs/python-ingestion.md— FastAPI pipeline and annotation contractdocs/data-model.md— MongoDB collections and S3 layoutdocs/ai-contracts.md— model usage, prompt boundaries, output contractsdocs/development-workflows.md— execution and verification playbookdocs/agent-playbooks.md— task-oriented lookup paths
MIT