NOHARM - analysis from released data

Reproduce the analysis of the First, Do NOHARM manuscript from released data: per-case score tables (the scientific data) plus scored-aggregate JSONs, panel-by-panel in analysis.ipynb as

1. prepare the data and estimand   2. analyze the result   3. plot the panel

Run

python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
jupyter notebook analysis.ipynb    # run all cells, top to bottom

Reusable statistics live in repro_stats.py (specialty-stratified case bootstraps, crossed random-effects, MCB + Holm top-tier brackets, B=10,000); plotting.py holds the shared theme and chart primitives. A full run takes well under a minute and recomputes every model-side panel from the released per-case tables.

What ships (contamination policy)

This repo contains no model responses, rubric text, case prompts, source case ids, source participant pseudonyms, participant demographics, or participant-level response rows. Case rows use stable cNNN pseudonyms consistently across the released model tables.

Per-case score tables (data/*.csv), all under fixed opaque ids that are consistent across tables (cNNN base case and sNN specialty stratum; the id maps never ship):

table	one row per	notes
donoharm-case-performance.csv	prompt x model x base case	default, concise, and thorough metrics, 3 dp
donoharm-perturbation-performance.csv	displayed model x base case x variant	S3e group and weighted F1, 3 dp
donoharm-gaming-case-performance.csv	contamination control x base case	S3f case split and weighted F1, 3 dp
donoharm-length-components.csv	displayed model x base case x variant	S6a response length and raw score components, 3 dp
donoharm-category-condition-performance.csv	displayed model x base case x variant	S6b weighted F1 under each category condition, 3 dp
donoharm-severity-components.csv	displayed model x base case x variant	S7 weighted-F1 numerator and denominator components
multiagent-case-performance.csv	(prompt, config, k) x base case	combos k=3, solos k=3/11, record counts
multiagent-variant-performance.csv	S5 configuration x base case x variant	exact aligned variant scores for composition and team size
severe-mode-counts.csv	(prompt, model) x base case	exclusive full-severe-mode counts

The two model-level CSVs use the exact 49-model Full (stats) cohort of the manuscript. Prompt codes match public MAST: default (unprompted), concise (under 200 words), and thorough (under 500 words).

Deliberately withheld: model responses, rubric text, option-level judgments, real case identifiers, participant-level response rows, participant demographics, and source participant identifiers. Under IRB protocol 85610, human-study results are released as anonymized aggregate findings pending an approved data-sharing modification. Released model-level numeric outputs are limited to the fields needed to recompute the manuscript panels. The study targeted at least 100 participants; 101 clinicians were analyzed.

A maintainer-side release gate enforces this policy before every sync: an explicit file manifest, per-file schema allowlists, cohort checks, and sweeps for source case ids, participant identifiers, credentials, and raw-data references; unregistered files fail the gate.

Figure map

figures.json is the machine-readable manifest (manuscript number + section) for the panels implemented in analysis.ipynb. The notebook previews each dataframe at its first use only. Later panels identify the reused input without printing the same preview again.

panel	plot input	status
3a	severe-mode-counts.csv	recomputed from model-case counts
3b, 3c	donoharm-case-performance.csv	recomputed from unprompted model-case scores
3d	donoharm-case-performance.csv	all three prompts and both four-model groups recomputed; JSON supplies group membership
4b, 4c	human-study.json	plots released aggregates; participant-level data withheld pending IRB data-sharing modification
4d	human-study aggregate and model-case CSV	physician side plots released aggregates; LLM estimates recomputed
4e	human-study.json	plots released aggregates; participant-level data withheld pending IRB data-sharing modification
5b	multiagent-case-performance.csv	recomputed from configuration-case scores
S3a	severe-mode-counts.csv	recomputed from full-match severe counts
S3b	severe-count and model-case CSVs	full-severe quartiles, mode shares, Spearman correlation, and crossed binomial analysis recomputed
S3c, S3d	donoharm-case-performance.csv	all three prompts recomputed
S3e	donoharm-perturbation-performance.csv	expected worst-case curves recomputed from per-variant scores
S3f	model-case and gaming-case CSVs	open and held-out means recomputed from case scores
S4a-S4d	human-study.json	plots released aggregates; participant-level data withheld pending IRB data-sharing modification
S5a, S5b	multiagent-variant-performance.csv	exact paired contrasts recomputed from aligned variant scores
S6a	donoharm-length-components.csv	raw and corrected model scores recomputed from response-level components
S6b	donoharm-category-condition-performance.csv	condition means and rank sensitivity recomputed from response-level scores
S7	donoharm-severity-stability.json and donoharm-severity-components.csv	plots aggregate 107-model grids; recomputes the 49-model cohort as a sensitivity check

Figures and panels not listed (schematics, and panels whose inputs carry no released data) are produced separately from the frozen Vega-Lite specs in the MAST release kit.

Per-case data & what it lets you do

The released ids are pseudonyms, not a guarantee of anonymity. Numeric score, response-length, and severity-component vectors can fingerprint a case and may be linkable when combined with auxiliary data from the same responses. The bundle provides no direct source-id mapping or clinical text. The S7 sufficient statistics disclose counts of eligible and matched actions by severity tier, but not option text or identifiers.

You can re-run the model-side stratified cluster bootstrap, reproduce model-side headline CIs and the MCB + Holm top-tier bracket, re-rank, and re-slice. Human-study analyses are reported from released aggregate fits. You cannot from this bundle alone refit participant-level human-study models or reconstruct case prompts, model or clinician responses, rubric option text, or a person's legal identity.

License

• Code (analysis.ipynb, repro_stats.py, plotting.py): MIT (see LICENSE)
• Data (data/, figures.json): CC-BY-4.0

Same scheme as the public NOHARM benchmark kit. Cite the NOHARM manuscript when using these data or figures.