Two models can score the same on every accuracy benchmark and think nothing alike. This is the fingerprint that tells them apart.
12-dimensional metacognitive fingerprint · deterministic yes/no probes · ~170 API calls per model · 33 frontier LLMs · no training
An accuracy benchmark measures what a model gets right. It does not measure how the model got there. So two models can land within a point of each other on every published number and still be built to think in completely different ways — and which one fits your task depends entirely on the difference the benchmark cannot see.
Take GPT-OSS. It ships in two sizes, 20B and 120B, a 6× gap. On the accuracy numbers, the choice looks obvious and the two look nearly identical:
GPT-OSS 20B vs 120B — a 6× parameter gap
on the accuracy benchmarks, near-identical:
MMLU 90.0 / 85.3 AIME (tools) 96.6 / 96.0 SWE-Bench 62.4 / 60.7
on the metacognitive fingerprint: d = 0.54 apart
— farther than a typical within-family pair, near the distance between labs
the tell — a compositional prompt (A and B, each trivial alone):
20B answers both, every time 120B misses roughly one in three
the 6× larger model is worse at exactly what cannot be decomposed
GPT-OSS 20B vs 120B, decomposed by dimension (d = 0.54; composition accounts for 42%). Inset: Gemma 3 12B vs 27B at d = 0.18, diffuse.
Same accuracy profile, dramatically different behavioral profile. And the deployment implication is real: run the 120B for the irreducible problems and the 20B for the compositional workload that dominates production traffic — a recommendation invisible to any leaderboard.
The benchmark is a 12-dimensional metacognitive fingerprint, built from deterministic probes with machine-verifiable scalar outputs. It does not score a model against an answer key. It resolves each model's relative position — the signal is the difference between how a model responds and how its peers respond to the same probes. The cohort is not the context; it is the evidence.
The base probes are trivial on purpose. Every frontier model knows the capital of France and that 2+2 is 4. The measurement is what happens to the process when the content is free: calibration, knowledge-boundary detection, perturbation sensitivity, self-monitoring, composition. When content drops out, process remains. A full profile costs about 170 API calls at pennies per model, and stabilizes at a median of 13 probes per dimension — it tells you when it has measured enough.
142 probes span twelve tasks in four tiers:
- Tier 1 · core metacognition (68) — calibration, knowledge-boundary, memorization-vs-understanding, error-detection-with-correction, self-consistency.
- Tier 2 · perturbation sensitivity (60) — twelve probes under semantic, syntactic, pragmatic, length, and adversarial rewrites.
- Tier 3 · monitoring & control (22) — a coherence gate (committed minus direct) and pressure robustness.
- Tier 4 · novel axes (22) — a composition index
acc(A∧B) / max(acc(A)·acc(B), ε), local-to-global coherence, and rule-out F1.
Five of the twelve dimensions saturate — every strong model maxes them, and a pair diverging on a saturated axis is itself a signal. The other seven carry continuous spread. Distances are cosine over z-scored dimensions, with anytime-valid Empirical-Bernstein confidence sequences deciding when a profile has converged.
The fingerprint generates hypotheses about how these models were built. The labs' own documentation confirms them — four for four.
- Scale is not a style. GPT-OSS 20B ↔ 120B sit at d = 0.54, 42% of it in composition. Gemma 3 12B ↔ 27B sit at d = 0.18, diffuse. Growing a family and re-architecting it leave different marks, and the fingerprint tells them apart. OpenAI's own card confirms distinct architectures.
- Post-training is phenotype-level, and it lives in the control axes. Qwen3-Next Instruct and Thinking share base weights, yet sit at d = 0.70 — and 94% of the split concentrates in four monitoring-and-control dimensions. The fingerprint resolves not just that two models differ, but where.
- Independent labs converge — as a distribution, not a cherry-pick. Of 452 cross-lab pairs, 75 (16.6%) sit closer than the within-family median. Gemini 3 Flash and GLM-5 are d = 0.18 apart; Claude Sonnet 4.6 and DeepSeek V3.2, d = 0.19. No shared architecture, weights, or lab — convergent evolution in behavioral phenotype.
- Version labels hide generational rewrites. Gemini 2.0 Flash Lite and 2.5 Pro sit at d = 1.65, the largest within-family distance in the cohort — and Google's report calls 2.5 a fundamentally new base with natively integrated thinking. Claude Opus 4.1 → 4.6 shows the same at d = 1.35.
The whole cohort at once: a 2D projection of the 12-dimensional fingerprint across 33 frontier models. Solid arcs mark cross-lab convergence; dashed arcs mark within-lab divergence.
A single-number benchmark works while models are far apart. As architectures approach general capability, that gap on any one accuracy axis collapses, and a lossy scalar loses its power to tell two strong models apart — the exact moment you most need to. The engineering structure behind the accuracy is still there; it has just moved to dimensions a leaderboard does not measure. A metacognitive fingerprint measures those dimensions directly, which is what makes it a framework that keeps working as the models get better.
The 142 probes, the twelve-dimension scoring, and the full analysis of all 33 models are in submission/ and notebooks/; the writeup with every figure is submission/writeup.md.
pip install -r requirements.txtRohan Vinaik — independent researcher. Built for the Measuring AGI competition (Metacognition track). When two models score the same, the difference is in how they think — so measure that.