examples: add the spa/direct arm of the tau2-airline onboarding

[aviweit]

What

Exports the live cap-evolve project as examples/skillberry_benchmarks_tau2_airline/spa/ —
the intervention: spa arm of that example's PROMPT.md. The capability is the airline
agent's tool surface only, and it reaches the model as a Skillberry Store skill that the
proxy injects into the agent's LLM calls, so the benchmark never sees skill files.

The spa/ subfolder keeps the prompt's one arm per run rule visible: the direct arm is a
separate onboarding in its own project and lands as a sibling, never merged into this one.
The two arms' numbers are not like-for-like and should never be averaged.

Contents

• capevolve.yaml + capevolve.smoke.yaml — the real and smoke run specs
• adapters/adapter.py + gateway.py — tasks/run_trials/score/trajectories/apply,
  and the OpenAI-compatible gateway wiring
• seed_capability/ — the generated seed: an intentionally empty my_skill/SKILL.md,
  14 agent-visible tool scripts, and the frozen primitive_tools/functions.py
• optimizer/INSTRUCTIONS.md (+ the prompt-only variant)
• split_ids.json (all 50 ids) and split_ids_task9.json / smoke_split.json
  (train = val = test = ["9"])

Both specs set protected_paths: ["primitive_tools/*", "my_skill/SKILL.md"]. Under SPA,
SKILL.md is the prompt-enrichment channel, so freezing it alongside the frozen substrate is
what makes this a genuinely tools-only capability.

Export changes

Two paths were rewritten so the export is portable; every other file is byte-identical
to the live project:

Field	Before	After
optimizer_instructions_file	absolute path	optimizer/INSTRUCTIONS.md (project-relative — also resolve_instructions_file()'s default)
runner_repo_path	../../vendor/…	../../../vendor/… (spa/ is one level deeper)

Both specs were re-read through cap-evolve's own spec reader afterwards: intervention,
skill_name, protected_paths, split files, and instructions file all resolve, and
runner_repo_path lands on the pinned vendor/skillberry-benchmarks checkout.

[aviweit]

Test: end-to-end optimization run on the SPA arm

Ran a full optimization through the intervention: spa arm of this example to verify the
delivery path works end to end, and to confirm the two properties that make the arm's
measurement meaningful. Numbers below are from that run's dashboard.

Configuration

Spec	capevolve.yaml (the one added in this PR)
Intervention	spa — candidate imported into the Skillberry Store, injected by the Skillberry Proxy-Agent
Capability	[tools], actions: [edit]
protected_paths	primitive_tools/*, my_skill/SKILL.md
Split	train = val = test = ["9"] (no-holdout; the engine logs a splits_warning and the test number is a fit metric)
Trials	5 per evaluation
Target model	gpt-oss-120b (tier mid)
Optimizer	claude-code, hill-climb, gate paired (k_se 0.2)

Result

Stage	val	Gate
Baseline (seed)	0.800 ± 0.200	—
cand_0001	0.800	rejected, Δ = +0.000
cand_0002	0.800	rejected, Δ = +0.000

Best candidate: seed — no candidate was accepted, so the champion never moved off the
seed. Sealed test: 0.200 ± 0.200 (per-trial rewards [0, 0, 0, 0, 1]). 2 iterations,
15 metric calls, optimizer cost $7.13 / 108,873 tokens; runner cost $0 (internal gateway).

Both iterations logged a gate_warning: with a 1-task split there is a single paired sample,
so the paired SE is 0 by construction and the gate falls back to STRICT (accept any Δ > 0,
reject 0). Both rejections came from that fallback. Note that val and the sealed test scored
the same unmodified seed capability on the same single task and returned 0.800 and 0.200 —
a 1-task/5-trial shape cannot resolve a Δ smaller than its own ±0.2, so this run exercises the
pipeline rather than establishing a capability result.

What I verified

1. The store is refreshed with the candidate's skill before every evaluation. The adapter's
apply() runs ahead of each evaluation and calls reset_store_to_skill(), which is
delete_skill → purge_orphans → upload_skill — a full replace from a clean slate, not a
merge. So no tool, snippet, or prose from the previous candidate can survive into the next
candidate's measurement; only the tagged frozen primitives are protected across the redeploy.
Confirmed for all three evaluations in the run (cand_0001, cand_0002, and the sealed
test FINAL).

Worth stating precisely: the refresh is per evaluation, not per iteration — which is the
stronger guarantee, since it also covers the baseline and the final sealed-test scoring.

2. The deployed skill's SKILL.md was empty — the capability really was tools-only.
my_skill/SKILL.md is 0 bytes in every candidate directory of the run (seed,
cand_0001, cand_0002), and protected_paths lists it alongside primitive_tools/*, so it
is outside the optimizer's edit surface. Under SPA, SKILL.md is the prompt-enrichment
channel, so an empty and frozen SKILL.md means the run carried no prose lever at all: the
only thing being optimized was the tool surface (my_skill/scripts/*.py), which is what this
arm is supposed to isolate.

---

Second test: 3-iteration run on the same arm

Repeated the test with a longer budget — cap-evolve run --max-iterations 3 --max-optimizer-usd 100
against the same capevolve.yaml. Same configuration as above (spa / tools-only / task-9
no-holdout split / 5 trials / gpt-oss-120b / hill-climb + paired gate), with a freshly
measured baseline rather than a reused one.

Result

Stage	val	Δ	Gate
Baseline (seed)	0.800 ± 0.200	—	—
cand_0001	0.600 ± 0.245	−0.200	rejected
cand_0002	0.600 ± 0.245	−0.200	rejected
cand_0003	0.600 ± 0.245	−0.200	rejected

Best candidate: seed. Sealed test: 0.400 ± 0.245 (per-trial [0, 0, 0, 1, 1]),
test_delta +0.000. 3 iterations, 25 metric calls, stall 3/3. Optimizer cost $6.93 /
100,291 tokens; runner cost $0. Wall-clock: 2,067 s optimizer + 2,873 s runner.

All three rejections came from the STRICT fallback (paired SE is 0 at n=1).

What the candidates edited, all inside my_skill/scripts/:

• cand_0001, cand_0002 — in-body eligibility guard in cancel_reservation.py
• cand_0003 — computed cancellation_eligible / cancellation_assessment fields added to
  get_reservation_details.py, plus the cancel_reservation.py guard

Same two properties verified again

1. Store refreshed with the candidate's skill before every evaluation — 5 evaluations in
this run (seed, cand_0001, cand_0002, cand_0003, test FINAL), each preceded by
apply() → reset_store_to_skill() (delete_skill → purge_orphans → upload_skill).

2. Tools-only capability — my_skill/SKILL.md is 0 bytes in all four candidate
directories (seed, cand_0001, cand_0002, cand_0003). The frozen substrate held too:
primitive_tools/functions.py is byte-identical to the seed in every candidate, so both
protected_paths entries were respected.

[skillberry-bot]

❌ Automatic Labeling Failed

An error occurred while trying to automatically label this pull request. Please check the workflow logs for details and add labels manually.

[OsherElhadad]

runner_repo_path is resolved as Path(project) / value (core/cap_evolve/specfile.py::resolve_project_path), where project is the deployed project dir passed via --project — and cli.py itself assumes workdir = proj_abs.parent.parent, i.e. the project dir is always exactly two levels below the repo root (.capevolve/project or, here, .capevolve-spa/project).

spa/setup.sh sets BASE="$REPO/.capevolve-spa" and PROJECT="$BASE/project" — also two levels below $REPO. So ../../../vendor/skillberry-benchmarks (three ../) resolves to a directory one level above the repo root, not to $REPO/vendor/skillberry-benchmarks where setup.sh actually clones the benchmark:

>>> from pathlib import Path
>>> p = Path('/repo/.capevolve-spa/project')
>>> (p / '../../../vendor/skillberry-benchmarks').resolve()
PosixPath('/vendor/skillberry-benchmarks')          # what this spec produces (wrong)
>>> (p / '../../vendor/skillberry-benchmarks').resolve()
PosixPath('/repo/vendor/skillberry-benchmarks')     # what setup.sh actually clones (right)

direct/capevolve.yaml uses ../../vendor/skillberry-benchmarks (two ../) for the identical project-dir depth, and that one is correct.

The PR description justifies the extra ../ as "spa/ is one level deeper" — but that's true only of the example source layout under examples/…/; it isn't true of the deployed project dir at runtime ($BASE/project is the same depth for both arms). cap-evolve check won't catch this either — pipeline_selftest.py passes runner_repo_path through unresolved just to check template placeholders; it never verifies the resolved path exists. So this silently hands the optimizer a --bench-repo pointing at the wrong (likely nonexistent) directory, degrading its read-only benchmark context with no error.

Same issue in capevolve.smoke.yaml:52.

[OsherElhadad]

Reviewed the diff, checked out the branch into a worktree, and traced the path-resolution logic against core/cap_evolve/cli.py / specfile.py. Summary:

Bug (see inline comment on spa/capevolve.yaml:51, also present in spa/capevolve.smoke.yaml:52): runner_repo_path: ../../../vendor/skillberry-benchmarks resolves to the wrong directory (one level above the repo root) given how spa/setup.sh actually lays out the project ($REPO/.capevolve-spa/project, same depth below $REPO as the direct arm's .capevolve/project). direct/capevolve.yaml's ../../vendor/skillberry-benchmarks is correct for the same depth. cap-evolve check won't catch this since pipeline_selftest.py never verifies the resolved path exists — it just checks template placeholders. Net effect: the optimizer silently gets no/wrong read-only benchmark context (task defs, real tool implementations, reward checks) during actual runs.

Structural asymmetry between the two arms: spa/ ships setup.sh + run.sh as the executable transcript of onboarding (per the file headers, these reproduce the intake/implement-and-check steps in one command). direct/ has no equivalent — no setup.sh/run.sh at all, so there's no reproducible way to stand up and run that arm short of manually re-deriving the onboarding steps from PROMPT.md. Worth adding for parity, or noting explicitly if it's intentionally out of scope for this PR.

Duplication: direct/adapters/adapter.py and spa/adapters/adapter.py share ~300 lines essentially verbatim — _STOP_LEAK_RE/_leaked_stop_continuation, _shown_metrics, _sim_to_rollout, score, and the whole gold-safe argument-level feedback stack (_iter_agent_tool_calls, _user_profile_facts, _localize_action, _localize_communicate, _derive_total_cost, _build_feedback). Given the PR's own stated goal of every-file-byte-identical-except-two-paths portability for examples, duplicating this much scoring/feedback logic across two "arms" of the same benchmark means any future fix to the feedback logic (e.g. the _localize_communicate heuristic, or the STOP-leak regex) has to be applied twice and will drift if someone updates only one. A shared module (even just within examples/skillberry_benchmarks_tau2_airline/) imported by both would remove the drift risk without hurting the "copy this example directory and it just works" portability goal.

Positives:

• Good discipline throughout on gold-safety: score()/_build_feedback() never read or echo the ground-truth values, only key names — well-guarded, and the module docstrings say why.
• The apply() guard-by-candidate-shape (checking for tools/tools.py vs my_skill/SKILL.md and naming the likely spec/seed mismatch in the error) is a nice fail-loud touch for what would otherwise be a confusing silent misconfiguration.
• gateway.py (both arms) is careful never to put api_key into llm_args, backed by a real assert in its own __main__ self-check — good, since trajectories()/store: git would otherwise commit it.
• The registry._domains monkeypatch in direct/adapters/adapter.py mirrors the existing pattern already used in examples/tau2_airline/adapters/adapter.py, so it's consistent with precedent in this repo (not a new global-state risk introduced by this PR), even though the base CapabilityAdapter docstring explicitly calls out apply()'s global-side-effect model as the reason materialize/live exists for concurrent-candidate evaluation. Not blocking, just flagging that this arm inherits that same single-global-slot constraint.

Not verified: I did not run an actual cap-evolve check/smoke run end-to-end — that requires cloning vendor/skillberry-benchmarks at the pinned commit, installing tau2-bench[skillberry], and live gateway credentials (OPENAI_BASE_URL/OPENAI_API_KEY), none of which are available in this review environment. The runner_repo_path bug above was verified by tracing the actual resolution code path and reproducing the path arithmetic locally, not by running the harness.