Paper Map

An interactive knowledge map that visualizes academic papers in 2D space, discovers research gaps, and provides AI-powered analysis — all in the browser.

What It Does

• Enter research keywords (e.g. "robot learning", "reinforcement learning robot")
• Automatically fetches papers from OpenAlex
• Computes semantic embeddings with SPECTER2, reduces to 2D via UMAP, clusters with HDBSCAN
• Renders an interactive heatmap in the browser — click anywhere to trigger AI analysis
• Three AI modes (works with any OpenAI-compatible LLM):
  • Discovery — find innovation opportunities in research gaps between clusters
  • Review — generate a literature review for the selected region
  • Mentor — multi-turn Q&A to help refine your research ideas

Quick Start

1. Clone

git clone https://github.com/huangyan28/paper_map.git
cd paper_map

2. Create a Python Environment

Python 3.10+ is required. We recommend conda:

conda create -n paper_map python=3.10 -y
conda activate paper_map

3. Install PyTorch

Install PyTorch separately to ensure proper GPU support for your hardware:

NVIDIA GPU (Linux / Windows):

pip install torch --index-url https://download.pytorch.org/whl/cu124

Apple Silicon (M1/M2/M3/M4 Mac):

pip install torch

MPS acceleration is included by default — no extra configuration needed.

CPU only:

pip install torch --index-url https://download.pytorch.org/whl/cpu

See pytorch.org/get-started for the full installation matrix.

4. Install Dependencies

pip install -r requirements.txt

5. Configure Your LLM

Paper Map works with any OpenAI-compatible API — OpenAI, Moonshot, DeepSeek, Ollama, etc.

cp .env.example .env

Edit .env:

LLM_API_KEY=sk-your-key-here
LLM_API_URL=https://api.openai.com/v1/chat/completions
LLM_MODEL=gpt-4o

Some common configurations:

Provider	LLM_API_URL	LLM_MODEL
OpenAI	https://api.openai.com/v1/chat/completions	gpt-4o
Moonshot	https://api.moonshot.cn/v1/chat/completions	kimi-k2.5
DeepSeek	https://api.deepseek.com/v1/chat/completions	deepseek-chat
Ollama (local)	http://localhost:11434/v1/chat/completions	llama3

6. Configure Your Research Domain

cp pipeline.example.yaml pipeline.yaml

Edit pipeline.yaml with your own keywords and filters:

search:
  queries:
    - "your research topic"
    - "another keyword"
  max_per_query: 500

filters:
  min_year: 2017
  min_citations: 50

embedding:
  device: auto    # auto | cpu | cuda | mps

device: auto automatically selects: NVIDIA GPU → cuda, Apple Silicon → mps, otherwise → cpu.

You can skip this step — the web UI will prompt you to configure on first launch.

7. Generate the Map

Option A: Command line

python -m scripts.pipeline

Option B: Web UI

python server.py
# Open http://localhost:8080
# A setup dialog will appear automatically if no data exists

8. Explore

conda activate paper_map
python server.py
# Open http://localhost:8080

• Click anywhere on the map → AI analysis of that region
• Left panel (Filters) → change keywords and click Rebuild Map
• Right panel → switch between Discovery / Review / Mentor modes

GPU Acceleration

The most time-consuming step is SPECTER2 embedding. Device auto-detection priority:

Hardware	Detection	Device
NVIDIA GPU	torch.cuda.is_available()	cuda
Apple Silicon	torch.backends.mps.is_available()	mps
Other	fallback	cpu

The SPECTER2 model (~440 MB) is downloaded on first run and cached locally.

Paper embeddings are also cached in SQLite — changing keywords won't re-embed previously seen papers, only new ones are computed.

Pipeline Stages

Stage	What It Does	Approx. Time
Fetch	Query OpenAlex for papers	~1-3 min
Embed	SPECTER2 semantic embeddings (cached)	First run: ~5-15 min (GPU) / ~30-60 min (CPU); incremental: seconds
Reduce	UMAP + HDBSCAN + KDE density estimation	~30 sec
Output	Generate map_data.json	instant

Each stage can also be run independently:

python scripts/01_fetch_papers.py
python scripts/02_embed_papers.py
python scripts/03_reduce_cluster.py

Project Structure

paper_map/
├── server.py                 # Local server (static files + LLM proxy + pipeline control)
├── pipeline.example.yaml     # Pipeline config template
├── requirements.txt          # Python dependencies (PyTorch installed separately)
├── .env.example              # Environment variable template
├── .gitignore
├── scripts/
│   ├── pipeline.py           # Unified pipeline entry point
│   ├── embedding_cache.py    # SQLite embedding cache
│   ├── config.py             # Shared paths & env vars
│   ├── 01_fetch_papers.py    # Stage 1: fetch from OpenAlex
│   ├── 02_embed_papers.py    # Stage 2: compute embeddings
│   └── 03_reduce_cluster.py  # Stage 3: UMAP + HDBSCAN + KDE
├── data/                     # Generated data (not committed)
│   ├── raw_papers.json
│   ├── embeddings.npy
│   ├── embedding_cache.db
│   └── map_data.json
└── web/                      # Frontend
    ├── index.html
    ├── css/style.css
    └── js/
        ├── app.js            # Main application logic
        ├── map.js            # ECharts map renderer
        ├── darkzone.js       # Gap analysis logic
        ├── llm.js            # LLM API client (OpenAI-compatible)
        └── wizard.js         # Pipeline controller