Research cut-off date: 2026-07-22. All statements in this repository are scoped to evidence published on or before this date. Model availability, benchmark leaderboards, and API prices change frequently. See Disclaimer.
A technical reference, comparative research survey, and reproducible evaluation framework covering contemporary proprietary and open-weight AI model families. Maintained by Ethical Tech CoLab.
Build status: Phase 1 (foundation) complete. Phases 2 to 5 are not yet written. See Repository status.
- Purpose
- Research questions
- Scope
- Research cut-off date
- Key findings
- Model families covered
- Benchmark categories covered
- Methodology summary
- Evidence-quality classification
- Compact comparison table
- Repository navigation
- Local installation
- Building the documentation
- Citing this repository
- Contribution rules
- Limitations
- Disclaimer
- Repository status
This repository exists to answer a practical question with academic rigour: given a defined task, a defined budget, and a defined deployment constraint, which AI model should be selected, and on what evidence?
Public discussion of model capability is dominated by provider launch materials and single-number leaderboard rankings. Those artefacts are frequently produced under evaluation conditions that are undisclosed, unreproducible, or not comparable across providers. This repository separates three categories of claim that are routinely conflated:
- Independently verified measurement.
- Institutional primary research and official technical disclosure.
- Provider-reported marketing and product documentation.
Every quantitative claim in this repository carries a source, a date, and an evidence grade. Claims that cannot meet that standard are recorded as Not publicly disclosed or Insufficient independent evidence rather than estimated.
The repository is intended for graduate researchers, procurement and policy analysts, machine learning engineers selecting production models, and sustainability researchers quantifying the resource footprint of inference.
The survey is organised around eleven research questions.
| ID | Research question | Primary chapters |
|---|---|---|
| RQ1 | How do contemporary model architectures differ, and which architectural choices measurably affect capability rather than efficiency alone? | 03, 05 |
| RQ2 | To what extent do published benchmark scores predict task performance under procurement conditions? | 09, benchmarks/ |
| RQ3 | How is factual accuracy measured, and how comparable are hallucination rates across model families? | 10 |
| RQ4 | Does reasoning-mode inference improve accuracy enough to justify its token and latency cost? | 06, 15 |
| RQ5 | How reliable are agentic and tool-use benchmarks as predictors of deployed agent reliability? | 07 |
| RQ6 | Does advertised context-window length correspond to usable retrieval and reasoning performance at that length? | 11 |
| RQ7 | How does tokenizer design distribute cost unevenly across languages and modalities? | 12 |
| RQ8 | What is the true cost per accepted task, as distinct from the advertised price per million tokens? | 15 |
| RQ9 | What is the energy, carbon, and water cost of inference, and what measurement conditions must accompany any such figure? | 16 |
| RQ10 | What hardware and memory conditions constrain self-hosted deployment of open-weight models? | 17 |
| RQ11 | Under what conditions do open-weight models substitute for proprietary models, and at what governance cost? | 18, 19 |
In scope. Text-primary foundation models and their multimodal extensions released or maintained by the model families listed in section 6; benchmark methodology and its limitations; inference-time performance, cost, and energy; hardware and memory requirements for self-hosting; licensing, privacy, and data-retention terms as published; and a decision framework for model selection.
Out of scope. Image-only and video-only generative models; speech synthesis models; embedding and reranking models except where they appear in retrieval-augmented pipelines; fine-tuning recipes for specific downstream tasks; training-infrastructure engineering; commercial vendor negotiation; and any forecast of unreleased models.
Boundary conditions. The repository does not run new evaluations of proprietary models against paid APIs as part of continuous integration. The evaluation framework in docs/evaluation/ specifies a reproducible protocol that a reader may execute independently, with results contributed back as Grade A evidence.
The repository-wide research cut-off is 2026-07-22.
Every file under docs/model-profiles/, docs/comparisons/, and docs/benchmarks/ carries its own cut-off line in a blockquote immediately below the title, because those files age at different rates. The file-level date takes precedence over the repository-level date. Dates are absolute and written as YYYY-MM-DD. Relative expressions such as "recently" or "the latest model" are prohibited by the style rules in CONTRIBUTING.md.
Each finding below answers a research question, rests on at least one Grade A or Grade B source, carries an absolute date, and is developed in exactly one chapter. Findings that would require ranking models are absent, because every benchmark result currently recorded has unstated evaluation conditions.
| # | Finding | RQ | Owned by |
|---|---|---|---|
| F1 | Inference optimisation changes energy use by a larger factor than model choice does. Fernandez and colleagues report reductions of up to 73 percent against an unoptimised baseline across hardware, serving frameworks, batching, and decoding strategies. Grade A. | RQ9 | 16 |
| F2 | Reasoning budget, not model size, dominates energy per query. Oviedo and colleagues estimate a median of 0.34 watt-hours for a representative frontier-scale query and 4.32 watt-hours at fifteen times the token use, roughly thirteen times higher. Grade A, analytical estimates rather than measurements of a named service. | RQ4, RQ9 | 16 |
| F3 | Advertised context capacity exceeds reliable capacity. Stanford's long-context evaluation states that support for long inputs does not imply long-context capability, and two independent benchmarks report degradation as context scales toward one million tokens. Grade A and Grade B. | RQ6 | 11 |
| F4 | Agents approach but do not reach human performance on interactive benchmarks. The AI Index reports a best model result of 66.3 percent on OSWorld against a human baseline of 72.35 percent, and 74.3 percent on WebArena against 78.24 percent. Grade B. | RQ5 | 07 |
| F5 | Hallucination persists in hard domains even with web access. HalluHard reports that the strongest web-enabled configuration tested still hallucinated on about 30 percent of difficult multi-turn conversations. Grade B. | RQ3 | 10 |
| F6 | Output tokens dominate cost. Across every price schedule recorded here, the output rate is several times the input rate, and output is generated sequentially, so verbosity drives cost, latency, and energy at once. Grade B. | RQ8 | 15 |
| F7 | A per-token price is not comparable across providers without a token-count measurement. Anthropic states that the Claude Sonnet 5 tokenizer can produce about 30 percent more tokens for the same text than its predecessor. Grade B. | RQ7, RQ8 | 12 |
| F8 | No published benchmark result in this survey can be used for ranking. All nine recorded results are provider-reported with unstated harness, sampling policy, and tool permissions. Grade C, recorded and labelled. | RQ2 | 09 |
Findings that would require energy, latency, or factuality datasets are not published, because those datasets are empty for the reasons given in 21. Research gaps.
Sixteen profile files, one per family, each following the fixed template defined in CONTRIBUTING.md.
| Family | Profile | Weights |
|---|---|---|
| OpenAI | openai.md | Closed, with open-weight releases |
| Anthropic | anthropic.md | Closed |
| Google Gemini | google-gemini.md | Closed, with open-weight Gemma line |
| Meta Llama | meta-llama.md | Open weights, community licence |
| DeepSeek | deepseek.md | Open weights |
| Mistral | mistral.md | Mixed open and closed |
| Qwen | qwen.md | Open weights |
| xAI Grok | xai-grok.md | Mixed |
| Microsoft Phi | microsoft-phi.md | Open weights |
| Cohere Command | cohere-command.md | Mixed |
| Amazon Nova | amazon-nova.md | Closed |
| AI21 Jamba | ai21-jamba.md | Mixed |
| Moonshot Kimi | moonshot-kimi.md | Mixed |
| MiniMax | minimax.md | Mixed |
| Zhipu GLM | zhipu-glm.md | Mixed |
| Other families | other-models.md | Mixed |
The Weights column above states the general posture of each organisation and is itself subject to verification during Phase 3. Per-model licence terms are recorded in data/models.csv with a source URL and are authoritative over this summary.
Nine benchmark files organised by construct rather than by vendor leaderboard.
| Category | File | Benchmarks documented |
|---|---|---|
| Overview and taxonomy | benchmark-overview.md | Cross-cutting: HELM, MedHELM, Chatbot Arena, MT-Bench |
| Knowledge and reasoning | knowledge-and-reasoning.md | MMLU, MMLU-Pro, GPQA Diamond, Humanity's Last Exam, ARC-AGI-1, ARC-AGI-2, ARC-AGI-3, FrontierMath, AIME, MATH, GSM8K |
| Coding | coding.md | SWE-Bench, SWE-Bench Verified, SWE-Bench Pro, Terminal-Bench, LiveCodeBench, HumanEval, MBPP |
| Agents and computer use | agents-and-computer-use.md | BrowseComp, WebArena, OSWorld, GAIA, AgentBench, BFCL, ToolBench |
| Multimodal | multimodal.md | MMMU, MMMU-Pro, MathVista, DocVQA, ChartQA, VideoMME |
| Long context | long-context.md | LongBench, LongCodeBench, RULER, MRCR, Needle-in-a-Haystack |
| Factuality | factuality.md | FACTS, SimpleQA, TruthfulQA, HaluEval, HalluHard |
| Multilingual | multilingual.md | Multilingual coverage and tokenizer-induced score distortion |
| Limitations | benchmark-limitations.md | Contamination, saturation, prompt sensitivity, sampling policy |
Every benchmark entry documents nine fields: what it measures, task format, scoring method, known limitations, contamination risk, whether tools are permitted, whether multiple samples are used, whether human baselines are comparable, and whether scores are suitable for procurement decisions.
The full protocol is in research-methodology.md. In brief:
- Data first. Machine-readable records in
data/*.csvare the single source of truth. Markdown comparison tables are generated from those records byscripts/generate_model_tables.pyandscripts/generate_benchmark_tables.py. A hand-edited comparison table without a corresponding CSV row is a defect and fails continuous integration. - Source before claim. Every row in
data/*.csvcarries asource_idthat resolves to a record indata/sources.csv.scripts/validate_sources.pyfails the build on an unresolved reference, a missing date, or a missing evidence grade. - Grade before comparison. Results are labelled with an evidence grade at the point of use. Results of different grades are never averaged or ranked against each other without an explicit warning in the surrounding text.
- Conditions before number. A benchmark score is recorded with its evaluation harness, sampling policy, tool permissions, and date. A latency figure is recorded with its percentile, concurrency, and region. An energy figure is recorded with the eleven conditions listed in docs/16-energy-use.md.
- Reproducibility. All scripts are deterministic, take file paths as arguments, depend on no paid API, and write outputs that are diffable in version control.
Three grades apply throughout the repository. The assignment rules and edge cases are defined in docs/appendices/source-quality-framework.md.
| Grade | Name | Admits | Typical use |
|---|---|---|---|
| A | Independently verified | Peer-reviewed studies; standardised academic benchmarks; reproducible third-party evaluations; independently measured system performance; public datasets with disclosed methodology | Any comparative claim between model families |
| B | Institutional primary research | University research reports; technical reports; model cards; benchmark methodology pages; official architecture disclosures | Architecture, context window, licence, and specification claims |
| C | Provider-reported | Launch benchmark tables; vendor latency claims; vendor customer studies; self-reported energy claims; product documentation | Recorded and labelled, never used alone to rank one provider above another |
Grade C evidence is included because excluding it would leave large gaps, particularly on pricing and on models whose weights are not public. It is always labelled as provider-reported at the point of use, and comparison tables carry an Evidence grade column so that a reader can filter to Grade A rows.
This table is generated from data/models.csv and data/pricing.csv by scripts/generate_model_tables.py --view compact. It is deliberately short. Full per-benchmark results live in docs/comparisons/, and full pricing lives in docs/15-token-economics.md.
| Provider | Model | Release date | Open weights | Context window | Input USD / 1M | Output USD / 1M | Evidence grade | Source date |
|---|---|---|---|---|---|---|---|---|
| Alibaba | Qwen 3.6 family | Not publicly disclosed | yes | 1000000 | Not publicly disclosed | Not publicly disclosed | B | 2026-07-22 |
| Alibaba | Qwen 3.7 family | Not publicly disclosed | yes | 1000000 | Not publicly disclosed | Not publicly disclosed | B | 2026-07-22 |
| Anthropic | Claude Fable 5 | Not publicly disclosed | no | 1000000 | 10.00 | 50.00 | B | 2026-07-22 |
| Anthropic | Claude Opus 4.8 | Not publicly disclosed | no | 1000000 | 5.00 | 25.00 | B | 2026-07-22 |
| Anthropic | Claude Sonnet 5 | Not publicly disclosed | no | 1000000 | 3.00 | 15.00 | B | 2026-07-22 |
| DeepSeek | DeepSeek V4 Flash | Not publicly disclosed | yes | 1000000 | Not publicly disclosed | Not publicly disclosed | B | 2026-07-22 |
| DeepSeek | DeepSeek V4 Pro | Not publicly disclosed | yes | 1000000 | 0.435 | 0.87 | B | 2026-07-22 |
| Gemini 3.1 Flash-Lite | Not publicly disclosed | no | Not publicly disclosed | Not publicly disclosed | Not publicly disclosed | B | 2026-07-22 | |
| Gemini 3.1 Pro | Not publicly disclosed | no | 1000000 | Not publicly disclosed | Not publicly disclosed | C | 2026-07-22 | |
| Meta | Llama 4 Maverick | Not publicly disclosed | yes | Not publicly disclosed | Not publicly disclosed | Not publicly disclosed | B | 2026-07-22 |
| Meta | Llama 4 Scout | Not publicly disclosed | yes | Not publicly disclosed | Not publicly disclosed | Not publicly disclosed | B | 2026-07-22 |
| Meta | Meta Muse Spark 1.1 | Not publicly disclosed | no | 1000000 | Not publicly disclosed | Not publicly disclosed | B | 2026-07-22 |
| Mistral | Mistral Medium 3.5 | Not publicly disclosed | yes | 256000 | Not publicly disclosed | Not publicly disclosed | B | 2026-07-22 |
| Mistral | Mistral Small 4 | Not publicly disclosed | yes | 256000 | Not publicly disclosed | Not publicly disclosed | B | 2026-07-22 |
| OpenAI | GPT-5.6 Luna | Not publicly disclosed | no | 1050000 | 1.00 | 6.00 | B | 2026-07-22 |
| OpenAI | GPT-5.6 Sol | Not publicly disclosed | no | 1050000 | 5.00 | 30.00 | B | 2026-07-22 |
| OpenAI | GPT-5.6 Terra | Not publicly disclosed | no | 1050000 | 2.50 | 15.00 | B | 2026-07-22 |
| xAI | Grok 4.5 | Not publicly disclosed | no | 500000 | 2.00 | 6.00 | B | 2026-07-22 |
The generator rewrites the content between the two markers and leaves the rest of this file untouched. Do not edit the region by hand.
| Path | Contents |
|---|---|
| docs/ | The handbook. Chapters 01 to 21, model profiles, benchmarks, comparisons, evaluation protocol, appendices |
| data/ | Machine-readable source of truth: models, benchmarks, pricing, context windows, energy studies, sources, and JSON Schemas |
| scripts/ | Validation and table-generation scripts. All are runnable from the repository root |
| notebooks/ | Analysis notebooks that read only from data/ |
| assets/ | Generated diagrams, charts, and table images |
| docs/interactive/ | The interactive companion page: a live cost-per-accepted-task calculator and four charts, each labelled with its evidence grade. Self-contained HTML, no build step, no external requests. Published at https://ethical-tech-colab.github.io/AI-Models-Research/interactive/ |
| research-methodology.md | Search strategy, inclusion and exclusion criteria, extraction protocol, and quality control |
| data-sources.md | The source register, the seed bibliography, and the outstanding verification queue |
| glossary.md | Terms and acronyms used across the repository |
| limitations.md | What this repository cannot establish, and why |
| references.bib | BibTeX database backing all numbered footnotes |
Start at docs/index.md or the full table of contents in SUMMARY.md.
Python 3.11 or later is required.
On macOS and Linux:
git clone https://github.com/Ethical-Tech-CoLab/AI-Models-Research.git
cd AI-Models-Research
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtOn Windows, using PowerShell:
git clone https://github.com/Ethical-Tech-CoLab/AI-Models-Research.git
cd AI-Models-Research
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txtRun the validation suite:
python scripts/validate_sources.py
python scripts/validate_tables.py
python scripts/validate_links.pyEach script exits with status 0 on success and 1 on any violation, and prints one line per violation in path:line: message form. No script fails silently.
The site is built with MkDocs Material. Configuration is in mkdocs.yml.
mkdocs serve # live preview at http://127.0.0.1:8000
mkdocs build --strict # fails on any broken internal link or navigation warningOn Windows, prefix with the activated virtual environment as shown in section 12; the mkdocs commands are identical.
mkdocs build --strict runs in continuous integration through build-docs.yml. A warning is treated as a failure, so a chapter listed in the navigation but missing from disk, or present on disk but missing from the navigation, blocks the merge. The three workflows are currently parked in .github/workflows-pending/ and are activated with one command; see that file for why.
Machine-readable citation metadata is in CITATION.cff. GitHub renders a formatted citation from that file in the sidebar.
Cite the repository, not an individual chapter, and include the commit hash and access date, because content changes:
@misc{etc_ai_models_research,
author = {{Ethical Tech CoLab}},
title = {{AI Models: Performance, Capabilities, Accuracy, Speed,
Energy Use, and Token Economics}},
year = {2026},
howpublished = {\url{https://github.com/Ethical-Tech-CoLab/AI-Models-Research}},
note = {Commit <hash>. Accessed YYYY-MM-DD.}
}When citing a finding, cite the underlying primary source recorded in data/sources.csv in preference to this repository. This repository is a survey and should not be used as the primary source for a result it did not produce.
Full rules are in CONTRIBUTING.md. The rules that cause the most rejected pull requests:
- No numerical claim without a citation resolving to
data/sources.csv. - No claim about a current model without an absolute date.
- Provider-reported results must be labelled Grade C at the point of use.
- No hand-edited generated table. Update the CSV and rerun the generator.
- No em dashes in prose.
- No repetition of an argument that another chapter already owns. Link to it.
- No blog post cited for an academic finding where a paper exists, no AI-generated summaries, and no search-result snippets.
- Benchmark scores obtained under different harnesses, sampling policies, or tool permissions are not placed in the same ranked column without a stated warning.
Use the issue templates for benchmark updates, model updates, and source corrections.
Stated in full in limitations.md. The principal constraints are that provider-reported evaluation conditions are frequently undisclosed and therefore unverifiable; that closed-weight models cannot be independently profiled for parameter count, memory footprint, or energy per token; that leaderboard and arena rankings are subject to prompt-selection and voting-population effects; that per-token energy figures are hardware-specific and cannot be generalised across serving stacks; and that this survey is not itself peer reviewed.
Benchmark rankings, arena positions, API prices, rate limits, context windows, and model availability change without notice and frequently. Every such value in this repository is a historical record of what a named source stated on a named date, not a statement about the present. Verify against the provider's current documentation before making a procurement, budgeting, or deployment decision.
Nothing in this repository is legal, financial, or procurement advice. Licence summaries are research notes, not legal interpretations; read the licence text.
| Phase | Contents | Status |
|---|---|---|
| 1 | Directory structure, configuration, schemas, methodology, source-quality framework, README, seed bibliography, validation scripts, continuous integration | Complete |
| 2 | Chapters 01 to 21 | Partial. Thirteen files written from registered sources: chapters 01, 02, 07, 09, 10, 11, 12, 14, 15, 16, 18, 20, 21, plus the bakeoff protocol and the benchmark glossary. The remainder carry a scope statement and a research checklist |
| 3 | Sixteen model-family profiles | Not started. Each file carries the fixed template and a research checklist |
| 4 | CSV population, validation, generated tables | Partial. 41 sources, 18 models, 8 pricing rows, 14 context-window rows, and 9 benchmark results are recorded; energy, latency, and factuality datasets are empty for stated reasons |
| 5 | Navigation completion, link and lint passes, citation validation, research completeness report, data-gap register | Partial. The full gate passes; the data-gap register is in chapter 21 |
Outstanding data gaps and unverified sources are tracked in data-sources.md.
Documentation and data are licensed under Creative Commons Attribution 4.0 International. Source code under scripts/ and notebooks/ is licensed under the MIT Licence. See LICENSE.