Skip to content

bench: record transfer-eval-8fold skill-transfer pilot (run 20260902001) - #459

Merged
OsherElhadad merged 1 commit into
skillberry-ai:benchmark-historyfrom
bcarmeli:bench-history/add-transfer-eval-8fold-run
Sep 7, 2026
Merged

bench: record transfer-eval-8fold skill-transfer pilot (run 20260902001)#459
OsherElhadad merged 1 commit into
skillberry-ai:benchmark-historyfrom
bcarmeli:bench-history/add-transfer-eval-8fold-run

Conversation

@bcarmeli

@bcarmeli bcarmeli commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Adds the missing dashboard record for the zero-shot skill-transfer pilot (transfer_eval_v1), whose raw results were already merged via #421 into skillsbench-history (results/transfer-eval-8fold/) but never surfaced on the benchmarks page.

  • 8 folds, 5 tasks, 2 domains. Each tasks[] row is one train→test fold (named <train>→<test>), not a SkillsBench task.
  • reward_baseline/reward_opt intentionally do not reproduce cap-evolve's own per-fold test_delta (always 0.0 by construction under --max-iterations 0 — the frozen donor skill is the only artifact evaluated, so it's compared against itself). Instead they show the comparison the source summary actually makes: the target task's own native seed vs. the transferred skill's score on it.
  • experiment: "transfer-eval-8fold" (using the tier/experiment split from bench: record skillsbench task-by-task-87 sweep #456/site(benchmarks): derive Benchmark/Type/Experiment filters, add Experiment column #457), tier: "full", iterations: 0, optimizer_model: "none" — no optimizer loop, pure evaluation.
  • benchmarks.json/meta.json regenerated via ci/benchmarks/lib/record.py aggregate.

Net effect across the 8 folds is a regression (0.2125 → 0.1625): transfer gives a small lift only where the native seed was near zero, and hurts where the target already had a strong seed. Directional only (small sample) — full detail in results/transfer-eval-8fold/summary.md on skillsbench-history.

8-fold zero-shot skill-transfer pilot (transfer_eval_v1, merged via skillberry-ai#421 into
skillsbench-history) had no dashboard record. Each row is one train->test
fold (not a SkillsBench task); reward_baseline/reward_opt show the target
task's own native seed vs. the transferred donor skill's score on it, per
the source summary's own comparison of interest -- cap-evolve's own per-fold
test_delta is always 0.0 by construction under --max-iterations 0 and would
be meaningless here.

Signed-off-by: Boaz Carmeli <boazc@il.ibm.com>
@OsherElhadad
OsherElhadad merged commit 671a39c into skillberry-ai:benchmark-history Sep 7, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants