diff --git a/docs/_includes/docs_sidebar.html b/docs/_includes/docs_sidebar.html index f7ed9b2990..a9250455ed 100644 --- a/docs/_includes/docs_sidebar.html +++ b/docs/_includes/docs_sidebar.html @@ -15,6 +15,7 @@
  • Importing Custom Modules
  • Adding New Models
  • Adding New Scenarios
    • +
  • HealthAdminBench
  • Adding New Tokenizers
  • Downloading Raw Results
  • Reproducing Leaderboards
    • diff --git a/docs/adding_new_scenarios.md b/docs/adding_new_scenarios.md index 1015baea82..21587ac136 100644 --- a/docs/adding_new_scenarios.md +++ b/docs/adding_new_scenarios.md @@ -19,7 +19,7 @@ For production MedHELM scenarios, see real implementations such as: - `medi_qa_scenario.py` — Hugging Face dataset with retry logic - `med_dialog_scenario.py` — external data download - `dischargeme_scenario.py` — summarization (requires `[summarization]` extra) -- `health_admin_bench_scenario.py` — computer-use HealthAdminBench wrap (see [HealthAdminBench](health_admin_bench.md)) +- `health_admin_bench_scenario.py` — computer-use HealthAdminBench wrap (see [HealthAdminBench](/health_admin_bench)) Run specs for MedHELM live in `src/helm/benchmark/run_specs/medhelm_run_specs.py`. diff --git a/docs/health_admin_bench.md b/docs/health_admin_bench.md index 77e5d56b14..43e62b1af2 100644 --- a/docs/health_admin_bench.md +++ b/docs/health_admin_bench.md @@ -7,22 +7,22 @@ HealthAdminBench is a **computer-use** evaluation of healthcare administration w Paper: [HealthAdminBench (Bedi et al., 2026)](https://arxiv.org/abs/2604.09937). Upstream harness: [som-shahlab/health-admin-bench](https://github.com/som-shahlab/health-admin-bench). -This page covers how to **clone the PacificAI forks**, **set the HAB path**, **install dependencies**, and **run** the scenario with `medhelm-run`. For general MedHELM install, see [Installation](/installation) and [Quick Start](/quick_start). API keys: [Credentials](/credentials). +This page covers how to **clone MedHELM and HealthAdminBench**, **set the HAB path**, **install dependencies**, and **run** the scenario with `medhelm-run`. For general MedHELM install, see [Installation](/installation) and [Quick Start](/quick_start). API keys: [Credentials](/credentials). ## What MedHELM runs One MedHELM instance is one Playwright episode. The adapter routes the outer request to the internal deployment `hab/harness`. OpenAI-chat-compatible `model_deployment`s run as `HelmBackedAgent` (each browser step is an inner `AutoClient` request). Native HAB agents (RandomAgent, Claude, Gemini) stay an override in `health_admin_bench_model_map.yaml`. The model completion is HAB `EvaluationResult` JSON. The leaderboard metric is `health_admin_bench_score` (subeval points / max, 0–1), under **Administration and Workflow**. -## Get the code (PacificAI forks) +## Get the code -You need **two** checkouts next to each other: MedHELM and HealthAdminBench. Use the PacificAI forks (this is where the MedHELM integration lives). Do not use the HAB upstream alone — MedHELM needs in-process hooks (`run_task(agent=..., llm_complete=...)`) that are on the fork. +You need **two** checkouts next to each other: MedHELM and HealthAdminBench. Clone HAB from [som-shahlab/health-admin-bench](https://github.com/som-shahlab/health-admin-bench) (`main`). MedHELM calls HAB in-process (`run_task(agent=..., llm_complete=...)`). ```bash # Parent directory for both repos (adjust as you like) mkdir -p ~/src && cd ~/src git clone https://github.com/PacificAI/medhelm.git -git clone https://github.com/PacificAI/health-admin-bench.git +git clone https://github.com/som-shahlab/health-admin-bench.git ``` Expected layout: @@ -30,7 +30,7 @@ Expected layout: ```text ~/src/ medhelm/ # PacificAI/medhelm - health-admin-bench/ # PacificAI/health-admin-bench + health-admin-bench/ # som-shahlab/health-admin-bench ``` The directories do not have to be siblings, but a single parent folder is easiest. Record the **absolute** path to the HAB checkout; MedHELM will not find task JSON files without it. @@ -296,7 +296,7 @@ helm-server --suite -o ./benchmark_output --port 8000 | Symptom | Fix | | --- | --- | -| Task JSON / `run.py` not found | Export `HEALTH_ADMIN_BENCH_ROOT` or pass `hab_root=`. MedHELM also looks at `./health-admin-bench` and `../health-admin-bench` relative to cwd (not the package install path). Use the PacificAI fork (`main`), not the som-shahlab upstream. | +| Task JSON / `run.py` not found | Export `HEALTH_ADMIN_BENCH_ROOT` or pass `hab_root=`. MedHELM also looks at `./health-admin-bench` and `../health-admin-bench` relative to cwd (not the package install path). Clone [som-shahlab/health-admin-bench](https://github.com/som-shahlab/health-admin-bench) (`main`). | | `ModuleNotFoundError: harness` / HAB imports | `uv pip install -e "$HEALTH_ADMIN_BENCH_ROOT"` into the MedHELM venv. | | Playwright browser missing | `source .venv/bin/activate && python -m playwright install chromium` (MedHELM venv). | | `The api_key client option must be set` / `openaiApiKey should be specified` | HelmBackedAgent reads `prod_env/credentials.conf`, not HAB `.env`. Match the key to the deployment prefix (`openaiApiKey`, `azureApiKey`, `stanfordhealthcareApiKey`). | @@ -318,7 +318,7 @@ helm-server --suite -o ./benchmark_output --port 8000 | Step | Command | | --- | --- | | Clone MedHELM | `git clone https://github.com/PacificAI/medhelm.git` | -| Clone HAB fork | `git clone https://github.com/PacificAI/health-admin-bench.git` | +| Clone HAB | `git clone https://github.com/som-shahlab/health-admin-bench.git` | | Point MedHELM at HAB | `export HEALTH_ADMIN_BENCH_ROOT=/absolute/path/to/health-admin-bench` | | Install HAB | `cd "$HEALTH_ADMIN_BENCH_ROOT" && uv sync && uv run hab install` | | Install MedHELM extra | `uv pip install -e ".[health-admin-bench]" && uv pip install -e "$HEALTH_ADMIN_BENCH_ROOT"` | diff --git a/docs/medhelm.md b/docs/medhelm.md index 69f0209370..777dc2e51a 100644 --- a/docs/medhelm.md +++ b/docs/medhelm.md @@ -123,7 +123,7 @@ Then open http://localhost:8000 in your browser. * **Schema:** Task taxonomy + metrics configuration powering `helm-summarize` and the UI. * **Release:** The release name of the leaderboard. -HealthAdminBench (computer-use healthcare admin workflows) is a separate scenario from OpenAI HealthBench. See [HealthAdminBench](health_admin_bench.md). +HealthAdminBench (computer-use healthcare admin workflows) is a separate scenario from OpenAI HealthBench. See [HealthAdminBench](/health_admin_bench). ---