Choose the acquisition function with the scientist instead of defaulting to expected improvement

[ronpandolfi]

Why

The skills presented the acquisition function as a lookup rather than a decision, and every path through them landed on "expected improvement":

• experiment-designer Step 1 asked a binary — "Exploration? Optimization? Both?" — whose optimization branch listed EI first. Scientists nearly always say "find the peak."
• EI had the broadest "Best for" cell in the built-in table ("Optimization (find max)"), while UCB's read like a knob you had to understand first.
• No skill listed a single EI failure mode, and none told the agent to involve the scientist in the choice.
• EI is the most-cited acquisition in the BO literature, so an LLM defaults to it. Silence was enough.

What changed

acquisition-functions/SKILL.md is restructured around choosing with the scientist: a decision table keyed on what they want to learn, an "Avoid when" column on the built-in table, and a caveats section for gpCAM-specific behavior the skills never mentioned:

• (a) EI clips improvement before forming z (surrogate_model.py:260), so wherever the posterior mean is below the incumbent it collapses to 0.3989·σ — identical in ranking to "variance". Measured: 193/200 grid candidates on a 15-point sin(x) fit, ratio exactly 0.398942. Filed separately as expected improvement clips improvement before forming z, degenerating to a scaled variance acquisition #55.
• (b) EI, PI, UCB, "maximum", KG, and NEI are all maximization-only and fail silently for minimization.
• (c) Plain EI/PI anchor to np.max(y_data), a noisy observation — which is what NEI exists to fix.
• (d) Multi-task plain EI sums stds across tasks and compares to a scalar max(y_data); KG/NEI scalarize the same objective correctly via _scalarized_blocks.
• (e) ask(n>1) silently rewrites string acquisitions to "total correlation".
• (f) "ucb"/"lcb" hardcode beta=3.0. Measured: implied β exactly 3.0000. This also corrects experiment-designer, which claimed the string exposed a tunable beta.
• (g) KG/NEI are the expensive built-ins — joint posterior per evaluation, KG loops per candidate.

Adds a radical_gradient recipe — sqrt(‖∇m‖)·σ — for mapping change through parameter space. Softening the gradient term lets uncertainty carry relatively more weight, so sampling spreads across every changing region instead of piling onto the steepest ridge. Built-in "gradient" stays the pick when there's one dominant transition.

Adds references/choosing-an-acquisition-function.md with a worked example per archetype: peak finding, mapping, change mapping, threshold, target value, minimization, batch, multi-task, cost-constrained.

Also registers the new troubleshooting and uncertainty-calibration skills in docs/source/claude-skills.md — they were added to CLAUDE.md but missed the docs table.

Verification

Claims (a) and (f) were measured against a live GP, and the radical_gradient recipe was run end-to-end through ask(). KG/NEI guidance is source-read only — I couldn't exercise it locally (installed fvgp is 4.6.5; the current code needs ~=4.8.1).

No library code is touched — docs and skills only.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G3DENMtUbk58gb5yVwEBBY

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.05%. Comparing base (3d576fa) to head (06bc0b6).

Additional details and impacted files

@@           Coverage Diff           @@
##           master      #56   +/-   ##
=======================================
  Coverage   90.05%   90.05%           
=======================================
  Files           9        9           
  Lines         945      945           
=======================================
  Hits          851      851           
  Misses         94       94           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.