tau2 airline — SPA variant (tau2_airline_spa)

[aviweit]

tau2 airline — SPA variant (tau2_airline_spa)

Adds a new end-to-end cap-evolve example that optimizes tau2 airline tasks with
LLM calls routed through Skillberry Proxy-Agent (SPA) + Skillberry Store,
instead of calling the upstream LLM directly.

What's new

File	Purpose
adapters/adapter.py	cap-evolve adapter wiring tau2's run_tasks to SPA
adapters/spa_env.py	SPA/Store service lifecycle helpers (start, stop, restart per candidate)
seed_capability/primitive_skill/	14 frozen primitive tools + SKILL.md (read-only reference for the optimizer)
optimizer/INSTRUCTIONS.md	Optimizer instructions scoped to CREATE-only composite skills
setup.sh / teardown.sh	One-shot setup and teardown of the full SPA stack
capevolve.yaml	Full run spec (50 tasks, 10 trials)
capevolve.smoke.yaml	Smoke spec (10 repr tasks, 3 trials, 1 iteration)
capevolve.quick-test.yaml	Quick-test spec (task 9, 1 trial, 1 iteration)
split_ids*.json	Task split files for full / smoke / quick-test / task-9 runs
PROMPT.md	Intake prompt for onboarding this benchmark from scratch

Key design points

• Frozen primitives, created composites. The optimizer can only create new
  composite skill packages alongside primitive_skill/. It cannot edit the seed.
• SPA restart per candidate. adapter.apply() uploads the new composite skill
  to the store and restarts SPA with SKILL_NAME=<candidate> before each eval.
• Credentials via env vars only. OPENAI_API_KEY / OPENAI_BASE_URL (or their
  IBM_* equivalents) — nothing hardcoded.
• Pinned deps. skillberry-benchmarks @ a3a8326, skillberry-store @ 0.2.1,
  skillberry-agent @ e359494.

How to run

bash examples/tau2_airline_spa/setup.sh

cp examples/tau2_airline_spa/capevolve.quick-test.yaml \
   examples/tau2_airline_spa/split_ids.quick-test.json \
   .capevolve/project/

SKILLBERRY_AGENT_DIR=vendor/skillberry-agent \
  SKILLBERRY_STORE_DIR=vendor/skillberry-store \
  PYTHONPATH=.capevolve/project/adapters \
  cap-evolve run --spec .capevolve/project/capevolve.quick-test.yaml

[skillberry-bot]

❌ Automatic Labeling Failed

An error occurred while trying to automatically label this pull request. Please check the workflow logs for details and add labels manually.

[OsherElhadad]

Thanks for this, @aviweit — this is a substantial piece of integration work and it's clear how much plumbing went into it. I reviewed the diff and then actually stood the whole stack up on a clean machine (real LLM via an OpenAI-compatible LiteLLM gateway, aws/gpt-oss-120b for both the agent-through-SPA and the user simulator, claude-opus-4-8 as optimizer) and drove cap-evolve run end to end. Notes below: what I liked, then what I found, roughly ordered by how much it matters.

What's good

• The integration itself is real and non-trivial. Four moving parts (Skillberry Store, tau2 Environment Manager, SPA, tau2's own run_tasks) wired into cap-evolve's adapter contract, and it does actually work — I got a real airline conversation routed agent → SPA → store → env manager → upstream LLM, scored by tau2's own reward. That is a lot of surface to get talking.
• Reproducibility is pinned properly. skillberry-benchmarks @ a3a8326, skillberry-store @ 0.2.1, skillberry-agent @ e359494 — commit/tag pins, not floating branches, and all three overridable by env var. This is better than most benchmark integrations.
• The adapter uses the contract the way it's meant to be used. run_batch + run_trials to let tau2's own batch runner parallelize instead of re-implementing trials; score() reads the reward out of rollout.metadata so it is genuinely deterministic on a fixed rollout; infra failures set Rollout.error instead of being scored 0.0; _shown_metrics() emits exactly one primary: true entry with db_match/cost_usd as display-only. That's the honest-eval contract respected in the right places.
• cap-evolve check stays offline. Lazy endpoint resolution in spa_env means the gate does no network I/O. Verified: {"ok": true, "stubs": [], "problems": []} with tasks('val') -> 50 and a deterministic scorer probe.
• The gold-safe feedback builder is the best part of the diff. _localize_action / _user_profile_facts / _derive_total_cost reconstruct what the agent itself observed and report the agent's own wrong argument values (payment_id=... (not on the user's profile; available=[...])) without ever leaking the gold value. That's exactly the kind of actionable-but-not-cheating signal the gate wants, and it's careful work.
• Credentials are env-only. Nothing hardcoded, and _upstream_llm_args() raises loudly rather than silently running against a default endpoint. setup.sh's _require_env preflight is a nice touch.
• optimizer/INSTRUCTIONS.md is well-written. The CREATE-only constraint, the two tool patterns with worked examples, the REAL/SAFE/VERIFIED gate on every design choice, the explicit non-overfitting rule, and the required PROCESS/JOURNAL handover — this is a genuinely good optimizer prompt.
• teardown.sh exists at all (neither tau2_airline nor skillsbench ships one), and setup.sh is staged, health-checked, and ends on the cap-evolve check gate. Good instincts.

What I ran

Step	Result
bash examples/tau2_airline_spa/setup.sh	got through steps 1–6; store came up at ~90s of the 100s budget, then segfaulted (see #13)
store + env manager + SPA restarted by hand (SPA repointed to :7002, see #3)	all healthy
cap-evolve check .capevolve/project	✅ {"ok": true, "stubs": [], "problems": []}
Run A — capevolve.quick-test.yaml as shipped (task 9, 1 trial, 1 iter)	baseline ✅ val 1.0; iteration ❌ died twice over — #6 then #1
Run B — same spec, 4 failing tasks, optimizer_usd_per_iter 15, SPA_PROVIDER_NAME/SPA_MODEL_NAME exported	✅ full loop: baseline val 0.000 → cand_0001 val 0.500, gate-accepted

Run B's accept line, for the record:

{"kind": "step", "candidate": "cand_0001", "accept": true,
 "reason": "paired Δ̄=+0.5000 > 0.0·SE=0.0000 (SE=0.2887, n=4)",
 "val": 0.5, "parent": "seed", "parent_val": 0.0,
 "optimizer_seconds": 794.11, "runner_seconds": 185.53, "opt_cost_usd": 9.404635}

Per task, tasks 11 and 19 flipped 0 → 1 and tasks 8 and 22 stayed at 0 — which matches what the optimizer's own PROCESS.md predicted, so the signal reaching it is real and not noise.

So the design is sound and the whole pipeline — eval, diagnose, propose, upload, restart, re-eval, gate — does work. Run A vs Run B is the diff between "works" and "doesn't": two config/wiring gaps, both fixable in this PR. Details below.

Blocking

0. The frozen seed mis-declares the primitive signatures, which artificially depresses the baseline. This is the one I'd fix first, because it affects what any number from this example means. tau2_primitive_functions.py types the structured arguments as str:

def book_reservation(..., flight_type: str, cabin: str, flights: str,
                     passengers: str, payment_methods: str,
                     total_baggages: 0, nonfree_baggages: 0, insurance: str): ...

tau2's actual airline tool is:

def book_reservation(..., flight_type: FlightType, cabin: CabinClass,
                     flights: List[FlightInfo | dict], passengers: List[Passenger | dict],
                     payment_methods: List[Payment | dict],
                     total_baggages: int, nonfree_baggages: int, insurance: Insurance): ...

Same for update_reservation_flights (flights: str) and update_reservation_passengers (passengers: str, whose own docstring says "An array of objects" directly under a str annotation). So the schema the agent is handed tells it to send JSON strings and free-text enums, the frozen primitive forwards them verbatim, and the write fails.

That's not theoretical — it's what I measured. Four tasks (8, 11, 19, 22), one trial: baseline val 0.000, all four "Database state does NOT match the expected final state". And the optimizer's own PROCESS.md independently diagnosed the same thing as its top-leverage cluster:

Cluster A — WRITE-ARG SERIALIZATION / ENUM (tasks 8, 11, 22). The agent passes list args as JSON strings and mangles enums ("basic economy", "one-way", insurance="none") … The frozen primitives forward these verbatim → store crashes ('str' object has no attribute 'flight_number') → the required DB write never lands.

Three of its five design choices (C1–C3) are pure argument-normalization wrappers. So the gain this example would report is substantially the optimizer repairing a transcription error in the seed, not a capability improvement — and since the seed is declared read-only, the optimizer can't fix it properly, only paper over it. Please bring the annotations in line with tau2's (list/int/the enum value sets, with the legal values in the docstrings). The baseline will rise, headroom will shrink, and what's left will be a real measurement. Also total_baggages: 0 / nonfree_baggages: 0 are literal-value annotations rather than int — they parse, but they flow into the generated tool schema.

1. The per-candidate SPA restart crashes the whole run. The optimizer created airline_policy_skill/, apply() called restart_spa(), and the run died:

File ".capevolve/project/adapters/spa_env.py", line 214, in start_spa
RuntimeError: SPA failed to start with SKILL_NAME=airline_policy_skill on port 7002

Three things stack up here:

• start_spa() doesn't reproduce the provider config setup.sh used. It forwards SKILL_NAME, USE_AGENT_TOOLS, USE_AGENT_PROMPTS, MCP_PROMPTS_POSITION, IBM_* — but not SPA_PROVIDER_NAME / SPA_MODEL_NAME, which setup.sh exports only inside its own shell. SPA therefore fell back to its default provider on restart: {"message":"access denied - RITS_API_KEY is not provided"}. Neither setup.sh's closing "Next:" hint nor the PR body's run command exports them, so the documented path hits this. Suggest env.setdefault("SPA_PROVIDER_NAME", "litellm") / SPA_MODEL_NAME in start_spa(), and adding them to the printed run command.
• stop_spa() leaves /tmp/skillberry-agent-service.pid behind. It force-kills by port with os.kill(pid, 9); the sentinel survives, and the next make run prints "skillberry-agent service is already running" and exits 0 without starting anything — so _wait_for_health burns 60s and raises. teardown.sh already knows to rm -f both sentinels; stop_spa() should do the same (I hit this by hand before the run, too).
• The RuntimeError propagates out of live() and aborts the run rather than degrading to a per-candidate infra error. One flaky restart shouldn't lose the whole budget — catching it and returning errored rollouts would let the harness treat it as noise (which it already knows how to exclude).

2. apply()'s "restart per candidate" isn't actually per candidate. The PR body and the adapter docstring both promise a restart before each eval, but:

skill_name = skill_dirs[-1].name
if skill_name != Adapter._current_skill_name:
    spa_env.restart_spa(skill_name)

_current_skill_name is a class attribute that lives for the whole process, and candidates inherit the parent's files — so if iteration 2 puts new content in the same directory name (very likely; the optimizer's own example name is airline_policy_skill/), the restart is skipped. Whether that yields a stale evaluation then depends on SPA's VMCP cache — it's keyed per env_id, so a new tau2 env may re-resolve the skill by name from the store, but a re-imported same-name skill's UUID/update semantics aren't something the adapter should be betting a reward number on. The cheap fix that removes the question entirely: make the uploaded skill name unique per candidate (e.g. <dir_name>__<candidate_id>), which makes the restart unconditional and the mapping candidate→SKILL_NAME injective. Related, in the same block:

• skill_dirs[-1] is alphabetically last, not newest (the comment says "newest") — with two composite skills present, which one gets evaluated depends on directory names.
• if not skill_dirs: return leaves whatever SKILL_NAME SPA last had. It happens to be right at baseline only because setup.sh started SPA with primitive_skill; on resume, on a re-eval of the seed, or on a candidate whose SKILL.md didn't get written (the detection predicate is (d / "SKILL.md").exists(), and I watched an iteration sit for a while with airline_ops_skill/scripts/ present but no SKILL.md yet — run 1's optimizer died from budget exhaustion mid-write) it silently scores the previous skill and attributes it to the new candidate. Worth setting SKILL_NAME explicitly on every apply(), and treating "candidate produced no valid skill" as an explicit error rather than a no-op.
• spa_env.upload_skill() returns a real bool (the store's import-anthropic does return success) but the return value is dropped — a failed upload evaluates the old skill and reports the score under the new candidate's name. Please raise on False.

3. os.kill(pid, 9) on whatever owns the port — and 7000 is taken on macOS. On this machine port 7000 is held by ControlCenter (AirPlay Receiver, on by default in recent macOS). So setup.sh/SPA can't bind, and worse, stop_spa() would have SIGKILL'd a system process. Two asks: scope the kill to a PID the adapter itself started (or at least verify the process is SPA before killing), and note the AirPlay conflict in the docs. Also, SKILLBERRY_AGENT_PORT is a dead knob — the port is hardcoded in three places outside this PR's control (tau2/config.py: SKILLBERRY_AGENT_URL = "http://127.0.0.1:7000", two literals in tau2/run.py, and uvicorn.run(..., port=7000) in SPA's main.py), so setting it changes the health check but not the routing. I had to patch the vendored copies to 7002 to test at all. Either drop the env var or document it as fixed at 7000.

4. LLM-authored code runs unsandboxed on the host. setup.sh starts the store with EXECUTE_PYTHON_LOCALLY=True, and actions: [create] means the optimizer writes arbitrary scripts/*.py that the store then executes in-process — in a process that holds OPENAI_API_KEY / IBM_THIRD_PARTY_API_KEY. The store already has a Docker execution path (file_executor.py falls back to it when the flag is unset). Given the code author is an LLM, I'd default to the Docker path and make local execution an explicit opt-in with a warning in PROMPT.md.

5. The guard-bypass claim doesn't hold as shipped. INSTRUCTIONS.md says "The agent sees ONLY the composite — cannot bypass the guard." After setup.sh, the store's primitive_skill has 15 tools, not 14 — _make_api_call is registered as a callable tool (it lives in scripts/make_api_call.py, and /skills/import-anthropic registers every public-ish function it finds):

['_make_api_call', 'book_reservation', 'calculate', 'cancel_reservation', ...]  # 15

_make_api_call(tool_name=..., **kwargs) is a universal escape hatch to any primitive, so Pattern 1 (WRAPPER) can't actually be enforced — and since the optimizer is told to copy the primitives (and therefore make_api_call.py) into its own skill, its composites inherit the hatch. Also, /tools/ ends up with 29 entries: setup.sh step 5 imports the 14 functions individually and then imports primitive_skill, which re-registers them all. The per-function loop looks redundant — the skill import alone covers it.

Medium

6. The quick-test spec can't do what it claims. It's described as "verify the full eval+optimize pipeline end-to-end", but:

• task 9 scores 1.0 at baseline → zero headroom, so the optimizer has nothing to fix and the iteration can only ever be rejected. The baseline phase's own job is to "confirm there is headroom to optimize"; please pick a task that fails at baseline (also note CI's integration-tests.yml already uses task 9 for the non-SPA variant).
• optimizer_usd_per_iter: 3.0 is below one opus iteration. Observed: terminal_reason: budget_exhausted, costUSD 3.3487975, "Reached maximum budget ($3)" — the optimizer died mid-iteration on the very first try, having produced a skill directory but no finished candidate. Raising it to $15 got the iteration past that point. ~$10–15 looks like a more realistic floor for this instruction set (it reads guidance/, primitive_skill/, and trajectories/ before proposing).

7. gate_k_se: 0.0 in the headline capevolve.yaml. With paired mode that accepts any Δ > 0, i.e. noise. HONEST_EVAL.md frames the k·SE bar as the thing that stops noise being mistaken for progress, and examples/tau2_airline uses 0.2 (your own smoke spec uses 1.0). 0.0 is defensible for the quick-test infra check; for a 50-task/10-trial publishable run I'd match tau2_airline.

8. No holdout in split_ids.json — train == val == test == all 50. Core notices and says so:

{"kind": "splits_warning", "msg": "test overlaps train/val (no-holdout fit) — the test number is NOT held out; report it as a fit metric"}

That's a legitimate deliberate choice (tau2_airline does the same) but it needs the same treatment docs/RESULTS.md gives it: label any number from this spec fit metric (no holdout) in a comment at the top of capevolve.yaml and in PROMPT.md. Ideally ship a genuinely held-out split file alongside, the way the 30/20 τ²-bench result did. Relatedly, split_train/val/test: 1.0 are dead lines when split_ids_file is set (and read as ratios summing to 3.0) — either drop them or comment that they're overridden.

9. Runner cost and tokens are always zero. From a real rollout: cost_usd 0.0, tokens 0. SPA doesn't report usage back to tau2, so agent_cost is 0 — which means max_usd: 200.0 can never bind on runner spend and the dashboard's cost panel will be blank for the interesting half. Worth either plumbing usage out of SPA or documenting explicitly that only optimizer spend is budgeted here.

10. The custom optimizer instructions can silently not apply. optimizer_instructions_file: examples/tau2_airline_spa/optimizer/INSTRUCTIONS.md is resolved cwd-relative first, then project-relative — and setup.sh never copies optimizer/ into .capevolve/project/. Run from anywhere but the repo root and core just omits --instructions-file and falls back to the generic scaffolded template, so the whole CREATE-only constraint quietly disappears and the optimizer believes it may edit the seed. Copying optimizer/INSTRUCTIONS.md into the project in setup.sh (and using a project-relative path) closes this.

11. teardown.sh is more destructive than its name suggests. It rm -rfs $REPO/.venv and all of $REPO/.capevolve — which is every other example's run artifacts and the scaffolded project, not just this example's. Please scope it to vendor/ + the services, or at least prompt before deleting .capevolve/. Same note for setup.sh's unconditional rm -rf "$PROJECT/seed_capability".

12. No smoke/run entrypoint, no docs, not on the site. Every other example ships run.sh (+ smoke.sh, and tau2_airline ships itest.sh wired into integration-tests.yml); this one ships none, so the PR body has to spell out manual cp steps. And the example is referenced nowhere outside its own directory — not in docs/OPTIMIZE_YOUR_OWN.md's "pick an example" table, not in docs/ADAPTER_TEMPLATES.md, and not in the GitHub Pages UI (site/optimize-your-own.html and site/results.html list toy_calc / tau2_airline / skillsbench only). Minimum I'd ask for: a run.sh + smoke.sh, a row in both the docs and site/optimize-your-own.html tables, and a short DEMO.md/REPRODUCE_tau2_spa.md once there's a real run to point at.

13. Setup is fragile on a cold machine. Two things I hit:

• wait_for_port gives the store 20×5s = 100s, but a first run does an npm install for the store UI; mine became healthy at ~90s. Give the store a longer budget (or wait on /health with a several-minute deadline).
• python3.11 -m venv .venv || python3 -m venv .venv silently falls back to whatever python3 is. Here that was 3.14, and the store then segfaulted mid-request (OMP: Error #179: Function pthread_mutex_init failed → exit code 139). Restarting with OMP_NUM_THREADS=1 worked around it, but the real fix is to fail loudly when 3.11 is missing rather than build an unsupported venv.

Small stuff

• _sim_to_rollout sets output=messages and trace=messages — the full conversation twice. One task/one trial was already 37 KB; at 50 tasks × 10 trials that's ~18 MB of duplicated JSON per eval. output could be a short summary.
• trajectories() returns the shared data/simulations/ dir, so when it's used it hands the optimizer every candidate's traces, unbounded. (In practice core prefers the per-tag rollout copy, so this override is mostly dead code — either scope it per candidate or drop it.) Also, that directory lives under vendor/, which teardown.sh deletes.
• _tee_to_log swaps the global sys.stdout and writes to /dev/tty specifically to defeat core's redirect_stdout. It's also not re-entrant/thread-safe. A plain log file (no tty, no global swap) would be less surprising.
• run_batch and run_trials are ~40 duplicated lines apart from num_trials; run_batch could be run_trials(..., n_trials=1).
• Dead imports in upload_skill (urllib.request, urllib.parse), and broad except Exception: pass in _load_env, stop_spa, upload_skill hides exactly the failures that produce a wrong number.
• total_baggages: 0 / nonfree_baggages: 0 in tau2_primitive_functions.py are literal annotations, not int — they parse, but they're wrong and they flow into the generated tool schema.
• make_api_call.py references a bare env_id that only exists because the store's executor prepends env_id = ... to the assembled module. That's a real (and clever) contract, but the file is un-importable standalone and it took me a while to confirm it wasn't a NameError — one comment would save the next reader the trip. Same file hardcodes http://127.0.0.1:8004 while setup.sh offers a TAU2_ENV_MANAGER_PORT knob.
• PROMPT.md line 168 documents cap-evolve run --split-ids-file ...; that flag doesn't exist (cap-evolve run takes --spec; the ids come from the spec's split_ids_file).
• split_ids_task9.json is byte-identical to split_ids.quick-test.json (same md5) and unreferenced by any spec — drop one.
• primitive_skill/SKILL.md: "Contains the the tools", trailing space after description:, no trailing newline at EOF.
• The store accumulates candidate skills forever. After my two runs, GET /skills/ shows ['airline_ops_skill', 'airline_policy_skill', 'primitive_skill'] — including the skill from the run that crashed. Since apply() resolves by name and SPA's resolve_skill_uuid searches the store by name, a re-uploaded same-name skill from a later run is at best ambiguous. Worth purging candidate skills at run start (setup.sh already knows how: DELETE /admin/purge-all) or namespacing them per run.
• optimizer/INSTRUCTIONS.md line 144 references `{{FAILURES}}` inside prose ("The {{FAILURES}} block below summarizes them"), and core substitutes every occurrence — so the whole failure summary gets inlined mid-sentence in a bullet and again at line 211. You can see it in the rendered prompt. It's inherited from examples/skillsbench/optimizer/INSTRUCTIONS.md:63, so worth fixing in both.
• INSTRUCTIONS.md shows another_skill/ ← you can create multiple but only one skill is ever made live (SKILL_NAME is singular, and apply() picks one). Worth saying "create exactly one" so extra dirs don't become dead weight in the diff.

Suggested order

If it were me: #0 first (it decides what any number here means), then #1 and #2 (the difference between "a run finishes" and "a run reports a trustworthy number"), then #6 so the shipped quick-test can actually exercise the loop out of the box, then #3/#4/#5, then docs + run.sh/smoke.sh + the site rows. Everything else is polish.

None of this changes the headline: the architecture works, and the loop closed with a gate-accepted 0.000 → 0.500 on a real model. Happy to re-run the whole thing once the restart path and the seed signatures are fixed — the stack is still up on my side.

[OsherElhadad]

Addendum — Run B's finalize step landed while I was writing the above, and it makes point #8 (no holdout) concrete enough that it's worth the extra comment.

Same four task ids in train, val, and test (as split_ids.json ships them), num_trials: 1:

split	reward	per task
val (cand_0001, gate)	0.500 (SE 0.289, n=4)	8: 0 · 11: 1 · 19: 1 · 22: 0
test (FINAL, sealed)	0.333 (SE 0.333, n=3)	8: infra-errored, excluded · 11: 0 · 19: 1 · 22: 0

Two things fall out of that:

1. The "sealed test" number is a re-roll of the val tasks, not a held-out measurement — and it came out lower than val (0.500 → 0.333) purely from run-to-run variance on identical ids. Task 11 went 1.0 → 0.0 on the same task. Anyone reading a baseline → test line from this spec would reasonably assume generalization was measured; it wasn't. Core does flag it (splits_warning: "report it as a fit metric"), so this is really just a request to carry that label into capevolve.yaml / PROMPT.md, and ideally to ship a genuinely disjoint split file alongside.
2. num_trials: 1 is doing real damage to the signal here. A single trial per task on a stochastic agent produced a full ±0.17 swing between two evaluations of the same candidate on the same tasks. Your full spec's num_trials: 10 is right; it's the smoke/quick-test specs where a 1-trial number will read as meaningful and isn't.

Credit where due on the same output: the infra failure was handled exactly per the contract — tau2 terminated for infrastructure reason: TerminationReason.TASK_FAILED set Rollout.error, and the harness dropped that trial from the mean (n=3) instead of scoring it 0.0. That's the adapter doing the right thing on the path that's easiest to get wrong.

[OsherElhadad]

Last addendum — the run finished, and the finalize output is more interesting than the val numbers, so I want to correct the impression my first comment may have left. It ended:

{"kind": "finalize", "test_reward": 0.3333, "test_baseline_reward": 0.5,
 "test_delta": -0.166667, "best_id": "cand_0001"}

The seed scored higher on test than the optimized candidate. Laid out per task (same four ids in train/val/test, num_trials: 1):

task	seed · val	cand_0001 · val	seed · test	cand_0001 · test
8	0	0	1	infra-errored, excluded
11	0	1	0	0
19	0	1	1	1
22	0	0	0	0
	0.000	0.500	0.500	0.333

The seed went 0.000 → 0.500 on identical tasks between its val and test evaluations — it passed tasks 8 and 19 on the second roll having failed both on the first. Task 19 is precisely the task the candidate's SKILL.md cancellation prose was written to fix, and the unmodified seed passes it too on a re-roll.

So the Δ̄=+0.5000 the gate accepted was substantially noise, and the honest sealed number is negative. Worth being precise about what that does and doesn't mean:

• It is not an algorithm bug. The gate did exactly what it was configured to do, and finalize faithfully reported a negative delta rather than hiding it — that's the honesty machinery working. The infra-errored task was excluded rather than scored 0.0, also correct.
• It is a spec-configuration problem, and it upgrades two of my earlier notes. At num_trials: 1, this variant's per-task variance is on the order of the effect being measured. gate_k_se: 0.0 then accepts it — and note that even 1.0 (your smoke value) wouldn't have caught it: +0.5 > 1.0 × 0.289. The bar that stops this is more trials, not a bigger k.
• Caveat on scope: the 4-task/1-trial split is my reduction for a fast wiring test, not your 50-task spec — and capevolve.yaml's num_trials: 10 is the right call. This is about the shipped capevolve.smoke.yaml (3 trials) and capevolve.quick-test.yaml (1 trial): your quick-test comment already says "Not a quality signal — purely an infrastructure/wiring check", which is exactly right, and this run is the empirical proof of it. I'd add the same warning to the smoke spec, and I'd resist ever publishing a number from either.
• And it sharpens feat(brand): add capybara logo to README and dashboard UI #8. With disjoint splits you would at least not be measuring the same four tasks twice and calling the second roll "held out". Here test_delta is a pure variance estimate dressed as a generalization result.

That's everything from my side. Net read unchanged: the integration is real, the loop closes, and the work needed is in apply()/start_spa() (#1, #2), the seed's argument types (#0), and the spec/doc hygiene around them — not in the design.

[aviweit]

@OsherElhadad , thanks a lot for the feedback.

I made changes fixing issues and addressing most of your comments. Can you please try it again?

I am aware of comment number 0. The primitive tools seed, including their signature, are based on the skillberry-skill-maker/skillberry-benchmarks (cc @eranra , @bcarmeli ). I think addressing it requires more design changes that we may want to discuss. Our first goal would be to integrate Tau2 SPA into Cap-evolve and to see that the optimization process properly use SPA as a black-box optimization approach.

Please find below my current settings I am using for the end to end runs.

Thanks !

Settings

.env

SPA_PROVIDER_NAME=litellm
SPA_MODEL_NAME=openai/aws/gpt-oss-120b

OPENAI_API_KEY=sk-****
OPENAI_API_BASE=https://ete-litellm.ai-models.vpc-int.res.ibm.com
OPENAI_BASE_URL=https://ete-litellm.ai-models.vpc-int.res.ibm.com

capevolve.yaml

# --- what to optimize -------------------------------------------------------
capabilities: [skill-package]            # the single airline_skill (tools + prompt enrichment via SPA)
capability_path: seed_capability         # holds airline_skill/ (editable) + primitive_tools/ (frozen)
actions: [modify]                        # optimizer MODIFIES the single airline_skill in place
skill_name: airline_skill                # the one skill in the store; never renamed
capability_sources: []

# --- who proposes edits -----------------------------------------------------
optimizer_skill: claude-code
optimizer_model: claude-opus-4-8
optimizer_max_turns: 100
optimizer_usd_per_iter: 40.0
optimizer_instructions_file: optimizer/INSTRUCTIONS.md   # project-relative — see setup.sh

# --- how to iterate ---------------------------------------------------------
algorithm_skill: hill-climb
algorithm_focus: all

# --- data + splits ----------------------------------------------------------
dataset_source: adapter
split_seed: 0
split_train: 1.0
split_val: 1.0
split_test: 1.0
split_ids_file: split_ids.json

# --- scoring ----------------------------------------------------------------
num_trials: 5

# --- acceptance gate --------------------------------------------------------
gate_mode: paired
gate_k_se: 0.0

# --- budget -----------------------------------------------------------------
max_iterations: 5
stall: 3
max_metric_calls: 0
max_usd: 200.0
max_optimizer_usd: 100.0

# --- iteration store --------------------------------------------------------
store: git

split_ids.json

{"train": ["9"], "val": ["9"], "test": ["9"]}

Invocation

teardown

bash examples/tau2_airline_spa/teardown.sh

note: does not touch .capevolve

setup

bash examples/tau2_airline_spa/setup.sh

run

Copy/paste the invocation command that setup.sh emits.

[aviweit]

Closing this PR as it is replaced by #409 as #424 and additional successive ones.