chi-bench: MedArise · openai/gpt-5.5

[jjxu217]

CHI-Bench submission

Domain: Prior Authorization (PA)
Submission: pa-codex-agent-gpt-5-5
Result: 17/25 pass@1 (68.0%)
Exceptions: 0
Validator: 0 errors, 0 warnings

This PR is narrowed to the PA submission only.

The PA agent uses case-agnostic operating guidance. It does not receive task-specific expected values, hidden rubric-derived fields, prior outcomes, or per-case answer hints. It derives codes, criteria, site, and terminal action from the benchmark task instruction, live MCP tools, exposed handbook/tool references, and current-run artifacts.

Implementation: pa-codex-agent uses PACodexAgentHarness, a thin wrapper around the upstream Harbor Codex harness. It installs/uses Codex in the sandbox, prepends case-agnostic provider-PA operating guidance, and delegates execution to Codex. All workflow mutations go through the official MCP tools.

[github-actions]

Submission validation — PR diff

✅ All checks passed.

[haolin-chen-actava]

Hi @jjxu217, thanks for the CHI-Bench submission.

Before we can merge, I ran some reproducibility + integrity checks and hit a
few things I'd like to sort out together.

Reviewing the trajectories, the task prompt is prefixed with an operating overlay that
contains case/task-specific ground-truth values matching the graded fields — e.g. procedure / diagnosis codes and dispositions on the provider side, and specific criterion IDs and expected observation values on the payer/CM side.
We believe this is information leakage: Per the benchmark rules, the agent should receive only the task instructions and the live MCP tools; it shouldn't be given task-specific answers or anything derived from the hidden expectations / rubrics. Could you remove the answer-bearing parts of the overlay and keep only general operating guidance?

A nit: We see the agent at ...:PACodexAgentHarness (and UMCodexAgentHarness /
CMCodexAgentHarness), but these aren't part of CHI-Bench. It would be great to include an implementation description in the PR.

We mark this PR as Request changes for now, feel free to have further discussion.

[jjxu217]

Thanks again for the review. I updated the PR to be PA-only, regenerated the packet, and added a short implementation description for PACodexAgentHarness in the PR body.

The current submission validates cleanly with 0 errors and 0 warnings.

[haolin-chen-actava]

In general LGTM. Quick one before merging: This packet was submitted twice — the earlier push reported ~52%, the current one 68% — and submission.json shows overall.n_trials: 50 vs n_tasks: 25. Did the 25-task suite get run twice? If so, would micro pass@1 (total passes / total trials) be more representative than the better single run? Thanks!

[jjxu217]

Thanks for catching this. I pushed a metadata-only correction for the aggregate overall.n_trials field in commit da18014d.

The current PA submission is a single regenerated PA-only full run from the updated/cleaned PACodexAgentHarness: 25 pa_provider tasks, 17 passes, 68% pass@1, 0 exceptions, and validator clean with 0 errors / 0 warnings. The per-domain row was already correct (n_trials: 25, n_tasks: 25, pass_at_1: 0.68); the stale overall.n_trials: 50 value was an aggregate metadata issue from packet generation, not evidence that the final 25-task suite should be scored as two combined runs.

I would not average in the earlier ~52% result because that was produced by an earlier harness attempt before the review-driven cleanup. The submitted 68% packet reflects the final agent harness configuration.

[haolin-chen-actava]

Thanks.