Does a single-cell foundation model add enough information to improve a real target-prioritisation decision?
FM Value Audit is a leakage-aware benchmarking framework for testing whether representations from models such as Geneformer and scGPT add measurable value beyond cheaper, easier-to-explain alternatives in immuno-oncology target prioritisation.
The project is intentionally designed to report negative results. A sophisticated embedding is not treated as useful merely because it looks biologically structured: it must improve a prespecified decision metric on unseen future outcomes.
Research software only. FM Value Audit evaluates methods for preclinical target prioritisation. It does not recommend a medicine, diagnose a patient or make a clinical decision.
v0.5.0a1 is a GitHub pre-release and execution alpha.
It contains two clearly separated evidence layers:
- Final v0.4 result: static Geneformer token geometry did not improve the prespecified AUPRC contrast over the historical baseline.
- Prospective v0.5 protocol: contextual cell-state embeddings are preregistered, but the GPU runs and future outcome evaluation are not yet complete.
The release must therefore not be described as evidence that contextual foundation-model representations improve target prioritisation.
Foundation models can produce impressive high-dimensional embeddings, but several questions are often left unanswered:
- Are the embeddings contextual, or only static token vectors?
- Were test outcomes used indirectly during feature selection or tuning?
- Does the model outperform a cheap historical or expression-based baseline?
- Is the apparent improvement larger than bootstrap uncertainty?
- Does the result transfer across datasets, diseases and experimental settings?
- Are unsuccessful methods still reported?
- Can every claim be traced to a sealed input, hash and frozen configuration?
FM Value Audit turns those questions into explicit software and statistical contracts.
The framework compares increasingly complex representations under the same split, target universe and evaluation rules:
flowchart LR
A[Historical target evidence] --> H[Matched prediction heads]
B[Popularity baseline] --> H
C[Manual expression features] --> H
D[Static Geneformer geometry] --> H
E[Static scGPT geometry] --> H
F[Contextual Geneformer states] --> H
G[Contextual scGPT states] --> H
R[Shape-matched random controls] --> H
H --> I[Gene-disjoint temporal evaluation]
I --> J[AUPRC, ranking metrics, calibration and uncertainty]
J --> K{Prespecified contrast supported?}
K -->|Yes| L[Evidence of incremental value]
K -->|No| M[Negative or inconclusive result retained]
The important unit is not embedding quality in isolation. It is incremental decision value over a realistic comparator.
The real temporal benchmark covered:
| Quantity | Value |
|---|---|
| Candidate genes | 17,527 |
| Solid-tumour indications | 23 |
| Development rows | 321,222 |
| Development positives | 42 |
| Final holdout rows | 79,542 |
| Gene-disjoint holdout genes | 3,476 |
| Final holdout positives | 14 |
| Paired bootstrap replicates | 2,000 |
The confirmatory contrast was frozen before the final holdout was opened:
AUPRC(geneformer_static_plus_cheap) - AUPRC(cheap_historical)
| Metric | Result |
|---|---|
| Geneformer static + historical AUPRC | 0.001980 |
| Historical baseline AUPRC | 0.002307 |
| Paired AUPRC difference | -0.0003276 |
| 95% paired gene-bootstrap interval | [-0.009485, 0.001392] |
| Confirmatory conclusion | NOT_SUPPORTED |
The interval crosses zero and the point estimate is negative. Static Geneformer token geometry therefore failed the prespecified test of incremental value.
scgpt_static_plus_cheap produced the highest observed point AUPRC, but it was a secondary method and its improvement interval against popularity crossed zero. It is not promoted post hoc into a positive result.
This is a central design principle of the project: the conclusion follows the frozen contrast, not the most attractive number found afterwards.
Before the temporal benchmark was available, the project tested a narrower melanoma-to-NSCLC response-expression proxy. It trained matched ridge heads on melanoma post-treatment response structure and applied the fixed heads to lesion-level NSCLC contrasts.
Static Geneformer geometry showed detectable structure in this proxy, but its interval against the manual-feature baseline crossed zero. This analysis is retained as supporting evidence only; it is not a clinical target-prioritisation result.
Important safeguards included:
- official lesion-level NSCLC response mapping;
- preservation of patients with discordant lesion responses;
- patient-cluster bootstrap resampling;
- matched random embeddings;
- separation of exploratory proxy claims from the final temporal benchmark.
The complete-cell Stage-1 layer processed three real single-cell datasets without cell subsampling:
| Dataset | Data retained | Biological role |
|---|---|---|
| GSE115978 | 7,186 cells; 23,686 genes; 33 samples | Melanoma raw-count contextual input |
| GSE120575 | 16,291 cells; 55,737 genes; 48 patient states | Melanoma TPM/manual-feature support |
| GSE179994 | 150,849 cells; 19,790 genes; 47 samples | Independent NSCLC raw-count contextual input |
Large matrices and checkpoints are not redistributed in the source release. Compact features, metadata, pseudobulk matrices, hashes and audit artefacts are versioned.
Version 0.5 asks a stricter question than v0.4:
Do contextual token states produced from real cells add prospective target-prioritisation value over a contemporaneous historical baseline?
The alpha provides:
- a frozen information cutoff of 4 August 2026;
- an outcome window beginning 5 August 2026;
- deterministic sample-context cell selection;
- equal-weight sample pooling rather than cell-count-weighted pooling;
- Geneformer and scGPT contextual extraction;
- identically shaped random controls using the same cells and pooling;
- nine preregistered biological context views;
- development-only dimensionality reduction;
- a label-only future power gate;
- score-freeze contracts that reject label-like columns;
- resumable CPU/GPU Colab notebooks;
- tamper and leakage-boundary tests.
The v0.4 labels have already been opened. They may inform development, but they cannot serve again as an independent test for selecting contextual methods. The v0.5 protocol therefore requires new outcomes after the cutoff and a new sealed evaluation.
| Milestone | Status |
|---|---|
| Protocol frozen | PASS |
| CPU preflight notebook | Ready to run |
| Geneformer contextual GPU notebook | Ready to run |
| scGPT contextual GPU notebook | Ready to run |
| Contextual development comparison | Waiting for GPU outputs |
| Scores frozen without future labels | Pending contextual outputs |
| First eligible future snapshot | 2027-02-04 |
| Final prospective efficacy conclusion | NOT_DUE |
The authoritative validation source is the live CI workflow, not a screenshot, PNG badge or manually edited pass counter.
Every push to main and every pull request executes:
| Executable gate | Live evidence produced |
|---|---|
| Test matrix on Python 3.11, 3.12 and 3.13 | JUnit XML and complete pytest log |
| Branch-aware coverage with an 85% minimum | coverage.json and Cobertura XML |
| Python compilation | Workflow command result |
| Model-registry validation | Full CLI audit log |
| Sealed temporal-benchmark validation | Full CLI audit log and explicit PASS_WITH_WARNINGS status |
| Synthetic benchmark from simulation to report | End-to-end workflow result |
| Ruff lint and formatting | Separate blocking quality job |
| Wheel and source build | Downloadable GitHub Actions artifact |
| Clean-wheel CLI and temporal smoke test | Package job result |
| Strict mypy | Blocking type-check job with its real workflow log |
The CI job also writes a machine-readable validation-summary.json containing the commit SHA, Python version, observed test count and observed branch coverage. These values are calculated from that workflow execution and are never represented by a fixed image.
Strict mypy is now a blocking gate. The contextual pandas and NumPy typing issues present in the uploaded kit were corrected before this repository repair.
For reference, the corrected source tree was validated locally with 70 passing tests, 86.78% branch-aware coverage, Ruff, strict mypy, compilation, model-registry validation and the temporal audit. These are historical repair measurements; the current repository state must be judged from the latest GitHub Actions run.
Run the same blocking controls locally with:
python -m compileall -q src tests
ruff check .
ruff format --check .
pytest --cov=fmva --cov-branch --cov-report=term-missing --cov-fail-under=85
fmva validate-manifest registry/models.yaml
fmva temporal-validate results/temporal_benchmark_v0.4
python -m buildStrict mypy is a blocking CI gate and currently passes on the corrected source tree.
See VALIDATION.md for the validation policy and scientific claim boundary.
git clone https://github.com/rsolerortuno/FM-Value-Audit.git
cd fm-value-audit
python -m venv .venv
source .venv/bin/activate # Windows: .venv\\Scripts\\activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"fmva --helpfmva temporal-validate results/temporal_benchmark_v0.4Expected status:
PASS_WITH_WARNINGS
The warning preserves an inherited 14-row Stage-2B accounting mismatch. Downstream sealed counts, hashes and evaluated rows remain internally consistent.
fmva simulate \
--output /tmp/fmva-synthetic.csv \
--genes 600 \
--seed 11
fmva benchmark \
--data /tmp/fmva-synthetic.csv \
--output /tmp/fmva-results \
--bootstrap 40 \
--seed 11
fmva report \
--results /tmp/fmva-results/benchmark_summary.json \
--output /tmp/fmva-report.mdRun in this order:
notebooks/v0.5/FMVA_v05_00_preflight_protocol.ipynb CPU
notebooks/v0.5/FMVA_v05_01_geneformer_contextual.ipynb GPU
notebooks/v0.5/FMVA_v05_02_scgpt_contextual.ipynb GPU
The notebooks are resumable at sample-context level. Detailed instructions are in notebooks/v0.5/RUN_IN_COLAB.md.
fmva validate-manifest Validate the model registry
fmva simulate Generate a controlled synthetic benchmark
fmva benchmark Compare representations and baselines
fmva report Render an auditable report
fmva temporal-validate Audit the sealed v0.4 temporal benchmark
fmva contextual-protocol Create/validate the v0.5 protocol
fmva contextual-validate Audit contextual embedding artefacts
fmva future-power-gate Check future labels without exposing scores
fmva freeze-future-scores Freeze predictions before opening labels
Use fmva --help for the exact command names and options in the installed build.
src/fmva/ production Python package
protocols/ frozen prospective v0.5 contract
notebooks/v0.5/ CPU/GPU contextual execution notebooks
results/temporal_benchmark_v0.4/ sealed real temporal benchmark
results/ood_proxy/ melanoma-to-NSCLC supporting proxy
results/stage1_full_cells/ compact complete-cell artefacts
reports/figures/ portfolio-ready result figures
reports/portfolio_summary.* concise machine-readable release summary
registry/ model availability and provenance
schemas/ manifest and contextual index schemas
tests/ unit, integration, tamper and leakage tests
docs/ methods, assumptions, decisions and runbooks
FM Value Audit is intended to demonstrate practical ability across both scientific ML and software engineering:
- temporal and gene-disjoint evaluation design;
- prespecified primary contrasts;
- rare-outcome metrics and paired bootstrap uncertainty;
- single-cell data processing at complete-cell scale;
- foundation-model checkpoint execution and random controls;
- lesion-aware response modelling;
- leakage auditing and claim boundaries;
- immutable hashes and cross-artifact validation;
- CLI and notebook interfaces;
- automated tests, coverage, linting, packaging and release manifests;
- transparent reporting of negative results.
- The final v0.4 holdout contains only 14 positive target-indication entries.
- Static token vectors are not contextual cell embeddings.
- The clinical OOD delta could not be computed because the melanoma holdout subset had zero positives.
- The supporting response-transfer analysis is an expression proxy, not target validation.
- Pretraining accession overlap remains unresolved.
- The contextual GPU notebooks have not yet been executed in this release.
- No future v0.5 outcome labels have been opened.
- The software cannot establish that a target is biologically causal, safe or clinically actionable.
- Execute contextual Geneformer and scGPT embeddings on the preregistered real-cell inputs.
- Compare pretrained encoders with shape-matched random controls and contemporaneous cheap baselines.
- Fit dimensionality reduction and prediction heads using development data only.
- Freeze prospective scores before accessing post-cutoff labels.
- Evaluate the first snapshot only when the frozen power gate is satisfied.
- Resolve strict NumPy typing debt and promote the software from alpha to beta.
- Expand the protocol to additional tumour types without reusing opened holdouts.
- Add contextual attribution analyses that separate cell state, disease and treatment effects.
The current scientific conclusion is deliberately modest:
Static Geneformer token geometry did not demonstrate incremental temporal target-prioritisation value over the historical baseline.
That negative result motivated the correct next experiment rather than a post hoc reinterpretation. Version 0.5 preregisters a stronger contextual-cell benchmark, matched random controls and a new prospective evaluation boundary.
The value of FM Value Audit is therefore not a claim that foundation models always work. It is a reproducible system for determining when their additional complexity is actually justified.
Machine-readable citation metadata are provided in CITATION.cff.
Released under the Apache License 2.0.