PaperTrail

https://autobenchmark.ai

Tracks benchmark citations across the LLM ecosystem - a heatmap sorted by benchmarks and models.

Scope

• Backfilled to 2025-09, then forward continuously.
• Labs: Big-5 (Anthropic, OpenAI, Google DeepMind, Meta, xAI) + open-weight leaders (DeepSeek, Qwen, Mistral, Moonshot, Z.ai). Onboard a lab by editing labs.yaml.
• Sources: primary sources only — release blogs, model cards, system cards. For open-weight models the Hugging Face README is accepted.

Data model

• data/registry.yaml — single source of truth for benchmark identity. raw name -> canonical aliases, plus type (agentic|chat) and domain.
• data/documents.yaml — one entry per source document (lab, model, container, url, pub_date, default_weight_class).
• data/extractions/<lab>/<id>.yaml — the benchmark mentions extracted from each document (raw name, optional weight_class override, score, model_config, methodology).
• data/citations.jsonl — generated by scripts/build.py; one record per (document, benchmark).

Source scoring — three distinct classes

Each mention is weighted by how it's cited, decoupled from the physical document (a single blog can carry both a headliner chart and a model-card comparison table):

class	points	what it is
blog_headliner	3	a chart or paragraph about a single benchmark in a release blog
model_card	2	a multi-benchmark comparison table in a standalone card or a table in a blog)
system_card	1	a reference in the long-form system-card paper

Pipeline

1. Discover (scripts/discover.py) — crawls each lab for new models (and blogs, model cards, system cards).
2. Extract benchmark mentions + methodology into data/extractions/ (scripts/extract.py.
3. Build scripts/build.py: citations to citations.jsonl.
4. Sync Harbor scripts/sync_harbor.py: fills harbor_* fields from harbor-framework/harbor registry.json + adapters/, fetched from Harbor main.
5. Heatmap scripts/build_heatmap.py → docs/index.html: the benchmark×model citation heatmap served at https://autobenchmark.ai.

Notes

• Unmatched benchmark names and unreadable/gated docs are flagged for human review as GitHub issues — never silently guessed.
• Raw source PDFs live in the companion private repo benchmark-citations-sources (Git LFS). Benchmark YAMLs are in this repo.