Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/_includes/docs_sidebar.html
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
<li><a href="{{ '/importing_custom_modules' | relative_url }}" class="{% if cur == 'importing_custom_modules' %}is-active{% endif %}">Importing Custom Modules</a></li>
<li><a href="{{ '/adding_new_models' | relative_url }}" class="{% if cur == 'adding_new_models' %}is-active{% endif %}">Adding New Models</a></li>
<li><a href="{{ '/adding_new_scenarios' | relative_url }}" class="{% if cur == 'adding_new_scenarios' %}is-active{% endif %}">Adding New Scenarios</a></li>
<li><a href="{{ '/health_admin_bench' | relative_url }}" class="{% if cur == 'health_admin_bench' %}is-active{% endif %}">HealthAdminBench</a></li>
<li><a href="{{ '/adding_new_tokenizers' | relative_url }}" class="{% if cur == 'adding_new_tokenizers' %}is-active{% endif %}">Adding New Tokenizers</a></li>
<li><a href="{{ '/downloading_raw_results' | relative_url }}" class="{% if cur == 'downloading_raw_results' %}is-active{% endif %}">Downloading Raw Results</a></li>
<li><a href="{{ '/reproducing_leaderboards' | relative_url }}" class="{% if cur == 'reproducing_leaderboards' %}is-active{% endif %}">Reproducing Leaderboards</a></li>
Expand Down
2 changes: 1 addition & 1 deletion docs/adding_new_scenarios.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ For production MedHELM scenarios, see real implementations such as:
- `medi_qa_scenario.py` — Hugging Face dataset with retry logic
- `med_dialog_scenario.py` — external data download
- `dischargeme_scenario.py` — summarization (requires `[summarization]` extra)
- `health_admin_bench_scenario.py` — computer-use HealthAdminBench wrap (see [HealthAdminBench](health_admin_bench.md))
- `health_admin_bench_scenario.py` — computer-use HealthAdminBench wrap (see [HealthAdminBench](/health_admin_bench))

Run specs for MedHELM live in `src/helm/benchmark/run_specs/medhelm_run_specs.py`.

Expand Down
14 changes: 7 additions & 7 deletions docs/health_admin_bench.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,30 +7,30 @@ HealthAdminBench is a **computer-use** evaluation of healthcare administration w

Paper: [HealthAdminBench (Bedi et al., 2026)](https://arxiv.org/abs/2604.09937). Upstream harness: [som-shahlab/health-admin-bench](https://github.com/som-shahlab/health-admin-bench).

This page covers how to **clone the PacificAI forks**, **set the HAB path**, **install dependencies**, and **run** the scenario with `medhelm-run`. For general MedHELM install, see [Installation](/installation) and [Quick Start](/quick_start). API keys: [Credentials](/credentials).
This page covers how to **clone MedHELM and HealthAdminBench**, **set the HAB path**, **install dependencies**, and **run** the scenario with `medhelm-run`. For general MedHELM install, see [Installation](/installation) and [Quick Start](/quick_start). API keys: [Credentials](/credentials).

## What MedHELM runs

One MedHELM instance is one Playwright episode. The adapter routes the outer request to the internal deployment `hab/harness`. OpenAI-chat-compatible `model_deployment`s run as `HelmBackedAgent` (each browser step is an inner `AutoClient` request). Native HAB agents (RandomAgent, Claude, Gemini) stay an override in `health_admin_bench_model_map.yaml`. The model completion is HAB `EvaluationResult` JSON. The leaderboard metric is `health_admin_bench_score` (subeval points / max, 0–1), under **Administration and Workflow**.

## Get the code (PacificAI forks)
## Get the code

You need **two** checkouts next to each other: MedHELM and HealthAdminBench. Use the PacificAI forks (this is where the MedHELM integration lives). Do not use the HAB upstream alone — MedHELM needs in-process hooks (`run_task(agent=..., llm_complete=...)`) that are on the fork.
You need **two** checkouts next to each other: MedHELM and HealthAdminBench. Clone HAB from [som-shahlab/health-admin-bench](https://github.com/som-shahlab/health-admin-bench) (`main`). MedHELM calls HAB in-process (`run_task(agent=..., llm_complete=...)`).

```bash
# Parent directory for both repos (adjust as you like)
mkdir -p ~/src && cd ~/src

git clone https://github.com/PacificAI/medhelm.git
git clone https://github.com/PacificAI/health-admin-bench.git
git clone https://github.com/som-shahlab/health-admin-bench.git
```

Expected layout:

```text
~/src/
medhelm/ # PacificAI/medhelm
health-admin-bench/ # PacificAI/health-admin-bench
health-admin-bench/ # som-shahlab/health-admin-bench
```

The directories do not have to be siblings, but a single parent folder is easiest. Record the **absolute** path to the HAB checkout; MedHELM will not find task JSON files without it.
Expand Down Expand Up @@ -296,7 +296,7 @@ helm-server --suite <suite> -o ./benchmark_output --port 8000

| Symptom | Fix |
| --- | --- |
| Task JSON / `run.py` not found | Export `HEALTH_ADMIN_BENCH_ROOT` or pass `hab_root=`. MedHELM also looks at `./health-admin-bench` and `../health-admin-bench` relative to cwd (not the package install path). Use the PacificAI fork (`main`), not the som-shahlab upstream. |
| Task JSON / `run.py` not found | Export `HEALTH_ADMIN_BENCH_ROOT` or pass `hab_root=`. MedHELM also looks at `./health-admin-bench` and `../health-admin-bench` relative to cwd (not the package install path). Clone [som-shahlab/health-admin-bench](https://github.com/som-shahlab/health-admin-bench) (`main`). |
| `ModuleNotFoundError: harness` / HAB imports | `uv pip install -e "$HEALTH_ADMIN_BENCH_ROOT"` into the MedHELM venv. |
| Playwright browser missing | `source .venv/bin/activate && python -m playwright install chromium` (MedHELM venv). |
| `The api_key client option must be set` / `openaiApiKey should be specified` | HelmBackedAgent reads `prod_env/credentials.conf`, not HAB `.env`. Match the key to the deployment prefix (`openaiApiKey`, `azureApiKey`, `stanfordhealthcareApiKey`). |
Expand All @@ -318,7 +318,7 @@ helm-server --suite <suite> -o ./benchmark_output --port 8000
| Step | Command |
| --- | --- |
| Clone MedHELM | `git clone https://github.com/PacificAI/medhelm.git` |
| Clone HAB fork | `git clone https://github.com/PacificAI/health-admin-bench.git` |
| Clone HAB | `git clone https://github.com/som-shahlab/health-admin-bench.git` |
| Point MedHELM at HAB | `export HEALTH_ADMIN_BENCH_ROOT=/absolute/path/to/health-admin-bench` |
| Install HAB | `cd "$HEALTH_ADMIN_BENCH_ROOT" && uv sync && uv run hab install` |
| Install MedHELM extra | `uv pip install -e ".[health-admin-bench]" && uv pip install -e "$HEALTH_ADMIN_BENCH_ROOT"` |
Expand Down
2 changes: 1 addition & 1 deletion docs/medhelm.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ Then open http://localhost:8000 in your browser.
* **Schema:** Task taxonomy + metrics configuration powering `helm-summarize` and the UI.
* **Release:** The release name of the leaderboard.

HealthAdminBench (computer-use healthcare admin workflows) is a separate scenario from OpenAI HealthBench. See [HealthAdminBench](health_admin_bench.md).
HealthAdminBench (computer-use healthcare admin workflows) is a separate scenario from OpenAI HealthBench. See [HealthAdminBench](/health_admin_bench).

---

Expand Down
Loading