Eval-env hygiene: persona-customer-support (extra_skill_paths) fires on "customer", derailing eval assertions

[lmorchard]

Summary

The dev/eval environment's resolved config includes extra_skill_paths that make personal persona-* skills real-discoverable — notably persona-customer-support, which the agent activate_skills whenever an input contains the word "customer". During #528 behavioral-suite authoring this repeatedly derailed routing/answer assertions (an unexpected activate_skill call breaks expect_no_tool / max_tool_calls bounds) and cost real tuning time across at least three suites (worked around by avoiding "customer" vocabulary in inputs).

Impact

Not a product defect — it's a property of this environment's resolved config bleeding into eval runs. But it makes evals authored in this environment fragile in ways that won't reproduce elsewhere, and quietly steers case wording.

Possible directions (not prescribing)

• Run evals with a clean/pinned skill set (exclude extra_skill_paths personal skills from the eval harness's resolved config), or
• Document a canonical eval config that doesn't inherit developer-personal extra_skill_paths, or
• At minimum, note the trigger-word hazard for future suite authors.

Context

Found via #528 behavioral-suite tuning; documented in docs/dev-sessions/2026-07-24-1618-eval-behavioral-diagnostics/notes.md (Deferred follow-ups).