Identifies potentially under-explored connections between research areas.
Give Aporia a topic. It fetches a paper corpus, embeds and clusters it, builds the citation graph, and surfaces pairs of related research clusters that few papers have actually cited across — ranked by a signal measured to correlate, modestly but really, with which pairs later get bridged by a real citation (see Does it work?). This is a citation-connectivity signal, not a claim about what the literature has or hasn't "thought of" — see Limitations below.
Aporia is a full research-gap-discovery pipeline wrapped in an institutional-looking web app. Given a topic, it:
- Fetches a paper corpus from Semantic Scholar (with an arXiv fallback), cached locally so repeat runs never re-hit the API.
- Embeds each paper with SPECTER2 (the proximity-tuned scientific-paper encoder), caching every embedding per-paper so a paper is only ever encoded once.
- Filters off-topic papers out by semantic similarity to the topic itself.
- Clusters the corpus (UMAP → HDBSCAN) into research sub-areas, each labelled with TF-IDF top terms plus a human-readable Gemini-generated name.
- Builds a directed citation graph across the corpus.
- Scores every pair of clusters for "gap-ness" — citation-sparse but semantically related pairs that also show open-problem language and a publication-time lag.
- Explains the top gaps with a Gemini-generated research question, testable hypothesis, and suggested method.
The result is a ranked list of research gaps, an interactive citation graph, and per-cluster publication-growth charts — all rendered in a dark "Ink Observatory" editorial theme (Cormorant Garamond + Inter).
- No API keys required to run the core pipeline. Semantic Scholar's public API and arXiv both work unauthenticated. Gemini features (advice, hypotheses, cluster names) are optional and degrade gracefully when no key is set.
- Cache-first. Fetched corpora and computed embeddings persist locally, so re-running a topic is near-instant and never rate-limited.
- Honest signals. Every score is normalized and explained in-app. An earlier version trained a GraphSAGE link predictor per job to estimate cross-cluster citation likelihood; it was removed after measurement showed its outputs were indistinguishable from chance on corpora of this size (a few hundred citation edges) — citation structure is now used directly, via a degree-normalized comparison against a null model.
Read this before treating any ranked gap as a claim about the literature itself, not just about this pipeline's view of it:
- Corpus-bounded citations. The citation graph only ever includes edges between
papers already in the fetched corpus (
limitpapers, one topic). A real citation to a paper outside that set is invisible to the null model — "under-connected in this corpus" is not the same claim as "under-connected in the literature." - Retrieval quality drives everything downstream. Clustering, the citation graph, and
every gap signal are only as good as what Semantic Scholar (or the arXiv fallback)
returned for the topic string. A narrow or ambiguous topic can return an off-target
corpus that clusters and scores just as confidently as a well-targeted one — see the
computational-paleography-of-historical-manuscriptsshowcase fixture for a real, unforced example of this (0.59 noise ratio, one visibly off-topic cluster). - Clustering is not perfectly reproducible run to run. Session 4's own sweep measured
ARI seed-stability as low as 0.25–0.63 across several real corpora at the clustering
configuration this pipeline uses — a different UMAP seed can move papers between
clusters, which moves which pairs get scored as gaps.
CLUSTERING_RANDOM_STATE=42is pinned specifically so this pipeline's own runs are reproducible; the instability is about how much a different seed would have found instead. - Low citation connectivity is not the same claim as "a research gap." A pair of clusters with a real deficit against the null model may be under-connected for a mundane reason (different subfields' citation conventions, a genuinely early-stage area, retrieval noise) rather than because nobody has thought to connect them. The backtest below measures whether flagged pairs get bridged more often than similar unflagged pairs — a real, positive, but modest-sample signal, not a validated causal claim about why a gap exists.
The public deployment runs Showcase mode only. Live queries are disabled there. To run a real, live query, run the project locally (see Installation).
The live deployment (Render backend + Vercel frontend) serves Showcase mode by
default — real, pre-saved results from previously completed pipeline runs
(data/showcase/*.json), rendered through the exact same UI as a live run, with no wait
and no compute.
Live queries (POST /api/query, the Live query tab) are turned off on the public
deployment because the full pipeline loads SPECTER2 and UMAP/HDBSCAN into memory
simultaneously during a run, which does not fit a free-tier RAM budget — running it
there would very likely crash the whole backend. This is a deliberate choice
to keep the demo free rather than pay for a larger instance. The gate is
ENABLE_LIVE_QUERY (see Environment variables); against the
public URL, POST /api/query returns 503 by design.
Aporia's frontend is a seven-module app (top nav). Every module renders through the same shared results components (gap list, citation graph, growth chart), so a showcase fixture, a live run, and a re-opened past run all look and behave identically.
| Module | What it does |
|---|---|
| Showcase | Pre-saved results from real completed runs — instant, no live compute. The public default. |
| Live query | Submit a topic, watch the pipeline run stage-by-stage, get ranked gaps. (Local only.) |
| Search | Semantic (cosine) search across every paper ever cached, across all topics. |
| Compare | Honest cross-topic semantic overlap between two already-run topics, cluster-by-cluster. |
| History | Re-open any past run's full results into the shared graph / growth / gap views. |
| How it works | A scrollytelling walkthrough of the whole pipeline, stage by stage. |
| Methods | The real backtest numbers (see Does it work?), cluster-quality stats, and the limitations above, in one page. |
Plus a header System dialog (cache/job stats, clear the paper cache) and a light/dark theme toggle.
aporia
├── core/ # the pipeline, one package per stage
│ ├── ingestion/ # Semantic Scholar + arXiv clients, corpus cache
│ │ ├── semantic_scholar.py
│ │ ├── arxiv_client.py
│ │ └── corpus_cache.py # cache-first SQLite store (kills repeat 429s)
│ ├── embedding/ # SPECTER2 encoder + SQLite vector cache
│ │ ├── specter_encoder.py
│ │ └── vector_store.py
│ ├── clustering/ # UMAP reduce → HDBSCAN → TF-IDF labels
│ │ ├── umap_reducer.py
│ │ ├── hdbscan_clusterer.py
│ │ └── cluster_labeler.py
│ ├── graph/ # directed citation graph + density metrics
│ ├── gap_detection/ # the four gap signals + the combiner
│ │ ├── citation_sparsity.py
│ │ ├── semantic_proximity.py
│ │ ├── temporal_lag.py
│ │ ├── future_work_miner.py
│ │ └── gap_scorer.py
│ ├── analysis/ # cross-topic comparison (Compare module)
│ ├── validation/ # offline backtest (gap signal vs. real citations)
│ └── llm/ # Gemini: advice, hypotheses, cluster names, topic fix
│
├── backend/ # FastAPI app wrapping the pipeline
│ ├── main.py # app + CORS
│ ├── routes.py # HTTP endpoints
│ ├── pipeline_runner.py # runs the full pipeline as an async job
│ ├── db.py # SQLite-backed job tracking
│ └── schemas.py # Pydantic request/response models
│
├── frontend/ # Vite + React + TypeScript
│ ├── public/
│ │ └── readme/
│ └── src/
│ ├── api/client.ts # the ONLY module that talks HTTP
│ ├── hooks/ # useJobPoller, useGraphData
│ ├── components/ # GapList, GraphViewer, GrowthChart, CompareModule, …
│ └── types/api.ts # mirrors backend/schemas.py
│
├── scripts/ # run_pipeline.py (the CLI), backtest.py, fit_gap_weights.py
├── tests/ # pytest suite for the signal math
├── docker/ # backend + frontend Dockerfiles
├── data/ # gitignored, except data/showcase/*.json fixtures
├── render.yaml # Render deploy blueprint
├── requirements.txt
└── docker-compose.yml
Data flow:
Semantic Scholar / arXiv
│ (cache-first)
▼
corpus cache ──► SPECTER2 embeddings (SQLite cache)
│
▼
relevance filter
│
▼
UMAP → HDBSCAN clusters ──► citation graph
│ │
└──────────┬─────────────┘
▼
five gap signals → gap_scorer → ranked gaps
│
▼
Gemini hypotheses
│
▼
FastAPI job API ──► React frontend
The pipeline can be run two ways: as scripts/run_pipeline.py (offline, one stage
at a time via --stage fetch|embed|cluster|graph|score, or all of them via --stage all, reading/writing data/{slug}/), or as a single async HTTP job through the FastAPI
backend. Both call the exact same backend.pipeline_runner stage functions, and share
the same embedding and corpus caches, so work done one way is reused by the other.
- Python 3.11+
- Node.js 18+
- ~2–4 GB disk for model weights (SPECTER2) and per-topic caches
- The first live run downloads SPECTER2 (~440 MB) from Hugging Face; it's cached afterward.
git clone https://github.com/yourusername/aporia.git
cd aporiapip install -r requirements.txt
pip install torch==2.12.1 --index-url https://download.pytorch.org/whl/cpu
torchneeds the CPU wheel index explicitly (Aporia is CPU-only — no CUDA anywhere).
cd frontend
npm install
cd ..Backend (starts the HTTP API):
uvicorn backend.main:app --reloadFrontend (separate terminal):
cd frontend
npm run dev # proxies /api/* to http://127.0.0.1:8000Open http://localhost:5173.
Or bring up both services together with Docker:
docker-compose up --buildEither through the Live query tab in the UI, or over HTTP:
curl -X POST http://127.0.0.1:8000/api/query -H "Content-Type: application/json" \
-d '{"topic": "Adversarial Robustness in Deep Learning", "limit": 200}'
# -> {"job_id": "<uuid>"}
curl http://127.0.0.1:8000/api/status/<uuid> # poll until "done" or "failed"
curl http://127.0.0.1:8000/api/results/<uuid> # 202 while running, else the resultThe first run of a topic fetches and embeds the corpus (a minute or two, depending on
limit); re-runs of the same topic are served almost entirely from cache.
A "gap" in Aporia is a relationship between two clusters — never a single paper or a single cluster. For every pair of clusters that's semantically related enough to be worth comparing, two signals combine into one score:
| Signal | What it measures |
|---|---|
| Under-connection | How far below a degree-preserving null-model expectation the observed cross-cluster citation count falls — a capped z-score against the null model's own variance, so a large deficit against a high-degree pair and a small deficit against a low-degree pair aren't conflated. The core gap signal. |
| Semantic proximity | How related the two clusters are, by cosine similarity of their (mean-centered) SPECTER2 centroids. |
| Future-work density | Fraction of papers on both sides whose abstracts contain "open problem" / "under-explored" language (regex-mined, verbatim). |
Combined with fitted weights, adopted from a logistic-regression fit against real post-cutoff citation outcomes (10 topics, 153 labeled cluster pairs, cross-validated AUC 0.755 — see does it work? below), not hand-set:
gap_score = 0.21 × (under_connection × semantic_similarity) # the core gap signal
+ 0.79 × future_work_density # direct "open problem" evidence
Two more signals — temporal lag (how far apart the two clusters sit in average
publication year) and size asymmetry (how lopsided the two clusters are in paper
count) — are still computed and shown in the UI as context, but no longer influence the
score. An earlier five-signal formula used a raw citation-density ratio
(citation_sparsity) in place of under-connection; measurement found it indistinguishable
from a constant across real corpora (standard deviation 0.01 across 21 real gaps in one
test corpus), so it was replaced by the null-model comparison above. Temporal lag and size
asymmetry were dropped from the score after the same audit found neither reliably
distinguished a real gap from noise, and a later fit (below) found neither would clearly
help even if re-added. The two remaining weights above started as an equal hand-set
70/30 split and were only replaced once a real fit cleared a strict pre-declared bar.
Pairs below a semantic-similarity floor are dropped entirely — a "gap" is only a meaningful claim between areas that are related enough for "there's a gap between these two" to make sense.
A real example, from the large-language-models showcase fixture: one flagged pair
observed 7 citations crossing between its two clusters where the degree-preserving null
model — "given how much each cluster cites and is cited overall, how many cross-cluster
citations would we expect if there were nothing special about this specific pair?" —
expected 18.1. That deficit, turned into a capped z-score and multiplied by the pair's
semantic similarity, is exactly what under_connection × semantic_similarity measures;
the app's gap-score explainer shows this same sentence ("N citations observed where E
would be expected") for every gap, not just this one.
An earlier version of Aporia also trained a small GraphSAGE link-prediction GNN per job to estimate each gap's link probability from the citation graph. It was removed after measurement showed its outputs were indistinguishable from chance on corpora of this size (a few hundred citation edges); citation structure is now used directly, via the under-connection signal above, rather than through a trained model.
Every one of these numbers is explained inline in the app (an "info" tooltip on each tile, a "How this is calculated" expander under each gap), so a non-specialist can see exactly what drove a score.
A backtest (full methodology and results) checks whether gap-flagged cluster pairs actually go on to get connected by a real citation more often than an equally-related, unflagged pair. On 8 cached topics with enough pre-cutoff data to test (cutoff year 2021), the current formula beats the one it replaced: pooled across all topics' pairs, gap-flagged pairs connected 88.9% of the time vs. 80.0% for the baseline (+8.9 points), while the old formula's own gap-flagged pairs did worse than its baseline (-9.1 points) on the same underlying data. Three of eight topics show a clean, decisive win for the current formula (as large as +66.7 points on one), one shows a real regression (-14.3 points), and four topics' corpora saturated to 100% connected either way, contributing no signal in either direction — so this is a real but still modest-sample result, not a settled one. Refitting the score's weights against this same evidence (10 topics, 153 labeled pairs) passed every pre-declared adoption check (AUC 0.755, well above chance) and did replace the hand-set 70/30 split with 21/79 — future-work language now carries most of the score. See the linked page for the full per-topic table, the cutoff-year sensitivity check, and the weight-fit adoption criteria.
The default landing view. Pick a pre-saved topic from the dropdown and its full results — ranked gaps, citation graph, growth charts — load instantly with no live compute. These fixtures are genuine, unmodified output from real completed pipeline runs, so what a visitor sees is exactly what the live pipeline produces.
The core output: research gaps ranked by gap_score, each card naming the two clusters
it bridges. The list is filterable — click a cluster in the citation graph and the gap
list narrows to gaps involving that cluster.
Each gap card shows the composite score as an animated meter, then breaks it into its contributing signals as a grid of stat tiles (each with its own tooltip and progress bar). A collapsible "Raw signals" section exposes the exact numbers and each cluster's representative papers (linked out to source). A Get advice button calls Gemini for a short, evidence-grounded suggestion, and the whole gap can be exported as a Markdown brief.
For the top-ranked gaps, Aporia asks Gemini for a concrete research question, a testable hypothesis, and a suggested method — rendered as a distinct "Suggested research direction" block, visually separate from the on-demand advice button. Without a Gemini key, this degrades to a documented "not available" state rather than erroring.
An interactive, force-directed citation graph (Sigma.js + graphology). Nodes are papers, colored by cluster and sized by citation in-degree, so influential papers read as bigger. A legend maps each color to its cluster label. Click any node to filter the gap list to that cluster. An arXiv-only corpus (no citation data) is flagged with a caution banner, since its graph-derived signals aren't meaningful.
Per-cluster publication counts over time (Recharts), for the largest clusters, so you can
see which sub-areas are emerging, stable, or declining. Cluster trend classification
(emerging / stable / declining) is computed for every cluster and surfaced on its
card.
Submit a topic and watch the job move through its stages — fetching → embedding → clustering → graph → scoring → done — polled live, then rendered into the same
results views as showcase. (Disabled on the public deployment; run locally.)
A global semantic search box: type a query and get the most cosine-similar papers across every topic ever cached — a different axis from any single run's results.
Pick two already-run topics and Aporia compares their clusters by embedding similarity only, using a shared joint-mean frame so the numbers reflect real overlap (not the near-constant ~0.9 that raw SPECTER2 cosine produces). Deliberately no cross-topic citation score — citations are in-set per topic, so that would be degenerate.
A two-column view: a list of every past completed run on the left, its full results on the right. Re-open any run into the shared graph / growth / gap views without re-running anything.
A centered scrollytelling narrative with a sticky pipeline stepper that lights up each of the seven stages as you scroll, plus a worked example thread. The in-app version of this README's pipeline explanation.
A header System dialog shows cache stats (corpus cache size, embedding count),
completed-job count, and completed-topic list, and can clear the paper cache. A
light/dark theme toggle (persisted in localStorage) switches between the dark "Ink
Observatory" and light "Parchment Journal" themes.
Fetched corpora are cached permanently in a local SQLite store
(data/global/corpus_cache.db), so re-running the same topic is served straight from
disk with no Semantic Scholar call — this is what avoids HTTP 429 rate-limiting on
repeat runs. The cache never expires on its own.
- Force a fresh fetch: send
{"refresh": true}toPOST /api/query, or pass--refreshtoscripts/run_pipeline.py --stage fetch. - Clear it: the System dialog's "Clear paper cache", or
POST /api/system/cache/clear— this also empties the separate Gemini response cache (data/global/llm_cache.db) in the same call, but never touches the jobs table or the embedding cache.
Embeddings are cached separately and per-paper (SQLite, data/global/embeddings.db),
keyed by (paper_id, model_version) — so a paper that appears in two topics is only ever
encoded once, and a fully-cached re-run never even loads the SPECTER2 model.
The backend serves a REST API under http://localhost:8000/api. Interactive docs
(Swagger UI) are at http://localhost:8000/docs while the server runs.
| Method & path | Purpose |
|---|---|
POST /api/query |
Submit a live pipeline job → { job_id }. Body: { topic, limit, refresh? }. |
GET /api/status/{job_id} |
Poll job status (pending…done/failed). |
GET /api/results/{job_id} |
Full results once done (202 while running). |
GET /api/jobs |
Paginated list of past jobs (limit, offset). |
GET /api/showcase |
List pre-saved showcase fixtures (instant, no DB). |
GET /api/showcase/{slug} |
One fixture's full results. |
GET /api/search?q=&limit= |
Cosine search across every cached paper. |
GET /api/compare?topic_a=&topic_b= |
Cross-topic cluster overlap. |
GET /api/gaps/{job_id}/{gap_id}/advice |
Gemini-backed suggestion for one gap. |
GET /api/system/stats |
Cache / job / embedding stats. |
POST /api/system/cache/clear |
Clear the corpus cache and the LLM response cache (never jobs or embeddings). |
GET /api/health |
Health check. |
Validation: topic must be non-empty and ≤200 chars; limit must be in [50, 800].
Violations return 422 with a typed ErrorResponse. Only one pipeline job runs at a
time — a second concurrent POST /api/query gets 429.
Rate limiting: a per-IP slowapi limiter guards the read-heavy routes —
/api/search is capped at 10/minute; /api/compare, /api/system/stats, and
/api/system/cache/clear at 30/minute each. A limit violation returns 429. This is
separate from the in-process job-slot concurrency guard on POST /api/query above,
which has no per-IP rate limit of its own.
| Variable | Where | Purpose |
|---|---|---|
SEMANTIC_SCHOLAR_API_KEY |
backend | Optional. Raises Semantic Scholar's rate limit for ingestion. |
LLM_PROVIDER |
backend | Optional, defaults to gemini. One of gemini | groq | none. Picks which backend powers gap advice, gap hypotheses, topic normalization, and cluster labels — see LLM provider below. |
GEMINI_API_KEY |
backend | Optional. Used when LLM_PROVIDER=gemini. Each feature degrades to a documented fallback if unset. Numbered keys (GEMINI_API_KEY_1..N) enable round-robin rotation across quotas. |
GROQ_API_KEY |
backend | Optional. Used when LLM_PROVIDER=groq. Same numbered-key rotation convention (GROQ_API_KEY_1..N). |
ENABLE_LIVE_QUERY |
backend | Optional, defaults on. Set false to disable POST /api/query (the public-deploy showcase-only gate). |
ALLOWED_ORIGINS |
backend | Comma-separated CORS allowlist. Falls back to localhost dev origins if unset — must be set to the live frontend URL in production. |
DATABASE_URL |
backend | Optional. Overrides the default local SQLite job store. |
VITE_API_BASE_URL |
frontend (build-time) | Backend origin for a deployed frontend build, e.g. https://aporia-backend.onrender.com. Leave unset for local dev/docker (uses the relative /api proxy). |
See .env.example for the full annotated list.
LLM_PROVIDER selects which backend powers the four LLM features (gap advisor, gap
hypotheses, topic normalization, cluster labels) without touching any code —
gemini, groq, or none. Each provider supports the same multi-key rotation
convention (GEMINI_API_KEY/GEMINI_API_KEY_1..N, or GROQ_API_KEY/GROQ_API_KEY_1..N),
and Groq is called directly via httpx against its OpenAI-compatible chat-completions
endpoint — no groq or openai SDK added. Gemini's free tier caps at roughly 20
requests/day/model, which makes regenerating the showcase fixtures across many corpora
slow; Groq's free tier is far more generous, which is why it exists here as an
alternative rather than a replacement.
none disables the LLM layer completely — no network call, no SDK import, every
feature returns its documented deterministic fallback. The public showcase
deployment sets LLM_PROVIDER=none (see render.yaml): it never calls an LLM at
request time, because every cluster label, gap hypothesis, and advisory note visible
on the live demo is already baked into the committed data/showcase/*.json fixtures
from when they were generated. This is worth stating plainly — it's a big part of why
the free deployment works at all with no LLM cost or quota risk in production.
Pure signal-math unit tests (no network / torch) live in tests/ and run fast:
pip install pytest
pytest- Backend → Render, as a Docker service (
docker/backend.Dockerfile, configured byrender.yaml), health-checked at/api/health.render.yamlsetsENABLE_LIVE_QUERY=falsefor the showcase-only public default. - Frontend → Vercel, native Vite build (no Docker), Root Directory
frontend/, withVITE_API_BASE_URLset to the live Render origin at build time.
Because the public deployment is showcase-only, it needs no GPU, no live-pipeline RAM,
and no paid tier — the pre-saved data/showcase/*.json fixtures are committed to the repo
and served straight off disk.
data/showcase/*.json are genuine, unmodified result_json blobs from real completed
runs. After a change to the scoring math, regenerate them so the public demo reflects
current behavior: run a live job locally for each showcase topic and save its result
(GET /api/results/{job_id}) to data/showcase/{slug}.json.
Pipeline: SPECTER2 (via transformers + adapters) · UMAP · HDBSCAN · NetworkX ·
PyTorch · SQLite · Google Gemini
Backend: Python · FastAPI · Uvicorn · SQLAlchemy · SQLite · httpx · slowapi (per-IP rate limiting)
Frontend: React 19 · TypeScript · Vite · Sigma.js + graphology · Recharts · Radix UI · Tailwind · Motion · Cormorant Garamond + Inter