results: transfer-eval-8fold zero-shot skill-transfer pilot report

[bcarmeli]

Adds results/transfer-eval-8fold/ recording the transfer_eval_v1 pilot: 8 folds over 5 tasks, each taking a frozen winning skill from one task and evaluating it zero-shot (--max-iterations 0, no adaptation) on a different task.

• summary.md — narrative report, following the summary.md convention of the sibling results/ folders
• transfer_eval_8fold.json — machine-readable per-fold record

Two things worth a second look

1. cap-evolve's own test_delta is 0.0 on all 8 folds — that's an artifact, not a finding. With --max-iterations 0 the transfer project's "seed" is the frozen donor skill, and it's the only artifact evaluated, so cap-evolve compares it against itself (best_id: "seed", test_reward == test_baseline_reward). Any real transfer effect has to be measured against the target task's native seed from results/results.json, which is what the report's transfer − native_seed column does.

2. Transfer actively hurts where the target already has a strong seed. weighted-gdp-calc seeds at 0.8 natively but drops to 0.2/0.3 under donor skills (−0.6, −0.5). The three positive lifts (+0.1/+0.2/+0.3) all land on targets whose native seed was 0.0–0.1 — the weakest possible baseline — and no fold approaches the target's own optimized score.

Results

#	skill from	evaluate on	job	status	transfer reward	native seed	native optimized	Δ vs native seed
1	shock-analysis-demand	shock-analysis-supply	555427	RUN	pending	0.0	0.2	pending
2	shock-analysis-demand	weighted-gdp-calc	540431	DONE	0.2	0.8	1.0 ‡	−0.6
3	shock-analysis-supply	shock-analysis-demand	543167	DONE	0.0	0.0	0.9	0.0
4	shock-analysis-supply	weighted-gdp-calc	540433	DONE	0.3	0.8	1.0 ‡	−0.5
5	weighted-gdp-calc	shock-analysis-demand	543168	DONE	0.0	0.0	0.9	0.0
6	weighted-gdp-calc	shock-analysis-supply	543169	DONE	0.1	0.0	0.2	+0.1
7	exam-block-sequencing	paratransit-routing	543170	DONE	0.2	0.0	1.0 ‡	+0.2
8	paratransit-routing	exam-block-sequencing	543171	DONE	0.4	0.1	1.0 ‡	+0.3

‡ in-loop val score — these four are KILLED_ceiling in results.json with final_test: null, so no held-out test eval ran.

Not yet final

Fold 1 is still running as LSF job 555427 and is marked _pending_ in both files. It completes the shock-analysis-demand ↔ shock-analysis-supply pair — the same-domain, most-natural-transfer cell, and the most interesting one in the table (its mirror, fold 3, came back 0.0). I'll push a follow-up commit to this branch with that number before merge.

Sample size is 8 folds over 5 tasks in 2 domains — directional only.

Related

The CCC/LSF operational lessons from this pilot (don't pass -W, poll cap-evolve.log instead of bjobs STAT, one dedicated host per job, -n 1) went to #412 instead, replacing its ## Batch (LSF) mode — TODO section, since that's where the CCC how-to lives.

🤖 Generated with Claude Code