docs(ccc): podman setup, CCC handoff, and skillsbench inventory

[bcarmeli]

Consolidated CCC/podman operational docs and the scripts/ccc/ helper scripts used to run cap-evolve on CCC compute nodes without root access: podman graphroot setup, job submission wrapper, and a smoke test, plus an inventory of what's staged for SkillsBench runs.

[skillberry-bot]

❌ Automatic Labeling Failed

An error occurred while trying to automatically label this pull request. Please check the workflow logs for details and add labels manually.

[OsherElhadad]

Review — docs(ccc): podman setup, CCC handoff, and skillsbench inventory

Reviewed the full diff (7 files, +2100/-0), read every added file end to end, syntax-checked and dry-ran the scripts locally, and ran the core suite.

Thanks for writing this up — the podman-without-subuid archaeology in CCC_PODMAN_SETUP.md is genuinely valuable, hard-won knowledge, and the "why" annotation on every workaround is exactly right. The troubleshooting table keyed to failure signatures is the best part of the PR. The SKILLSBENCH_INVENTORY.md numbers all check out (categories sum to 87; the 21-task coverage table has 21 rows; 43×3×7 = 903).

I'm requesting changes: the scripts as committed only work from one person's account, and CI is red.

---

Blocking

1. scripts/ccc/setup_podman.sh is dead code — nothing sources it.

Both runner scripts source the author's personal copy on shared storage instead of the sibling script this PR adds:

run_ccc_experiment.sh:179   source /dccstor/knewedge2/boazc/workarea/python/setup_podman.sh
run_ccc_smoke.sh:90         source /dccstor/knewedge2/boazc/workarea/python/setup_podman.sh

setup_podman.sh's own usage comment (line 6) points at the same external path. So for anyone who isn't boazc, the committed script is never executed, and both runners fail at Phase 1. Please use "$SCRIPT_DIR/setup_podman.sh" (SCRIPT_DIR is already computed at line 114 / 62). Same for the doc: CCC_PODMAN_SETUP.md and HANDOFF_CCC.md tell readers to source /dccstor/knewedge2/boazc/... and never mention that the script now lives in the repo — which is the single most useful thing this PR added.

2. run_ccc_experiment.sh:248 hardcodes the cap-evolve binary to one account.

CE_BIN="/dccstor/knewedge2/boazc/workarea/python/skillberry_ai/cap-evolve/.venv/bin/cap-evolve"
if [[ ! -x "$CE_BIN" ]]; then echo "FATAL: ..."; exit 2; fi

Hard exit 2 for every other user. Suggest CE_BIN="${CE_BIN:-$(command -v cap-evolve)}" with the current path as a documented fallback, or $PROJECT_ROOT/.venv/bin/cap-evolve.

3. submit_ccc_experiment.sh --dry-run is broken on any machine but the author's. I ran the exact invocation from the script's own header comment (line 19-20):

$ bash scripts/ccc/submit_ccc_experiment.sh --suite-id test --max-iterations 0 --dry-run
mkdir: /dccstor: Read-only file system
rc=1

Two defects in one line: CCC_LOGS defaults to /dccstor/knewedge2/boazc/ccc_logs (line 50), and mkdir -p "$CCC_LOGS" (line 90) runs unconditionally before the --dry-run guard at line 123. Please move the mkdir after the dry-run exit and default CCC_LOGS to something account-neutral ($PROJECT_ROOT/results/.ccc_logs, or require the env var).

4. CI is red — please fix before this lands.

• Docs link check fails on 2 dangling links, both in HANDOFF_CCC.md: [.env](.env) at line 73 (no such file, and there never will be — .env is gitignored) and the root-relative /u/boazc/.local/share/uv/... link at line 164.
• DCO fails — neither commit (1bf7167, 94aba9d) carries a Signed-off-by: trailer. git rebase --signoff on the two commits.

5. results/ is not gitignored, and both runners write into it. run_ccc_experiment.sh:128 and run_ccc_smoke.sh:65 write $PROJECT_ROOT/results/<suite>/<run>/ — including env_snapshot.txt (a redacted .env) and bench.log. A git add -A after a run commits credentials-adjacent artifacts. Please add /results/ to .gitignore in this PR.

---

HANDOFF_CCC.md — I don't think this belongs in the repo as-is

This reads as a paused-session journal rather than documentation, and it is internally contradictory. The header says "plumbing validated end-to-end … a one-task bench eval run smoke completes with errored=0" and "All CCC-specific workarounds are in place", while §"Where we paused" says "Result: still errors=1", §"For a fresh Claude session" says "The blocker is at the aardvark-dns/systemd layer", and the "Layers resolved" list says aardvark was resolved (item 5). A reader can't tell which is true.

Also:

• It is addressed to an AI session ("For a fresh Claude session picking this up", "You do NOT have SSH access… ask the user to paste commands' output", "Good luck.") rather than to a contributor.
• Ground truth is one specific person's account: boazc, /dccstor/knewedge2/boazc/…, podman-561567, host cccxc554, UIDs 561567:608693.
• §"Error log archive" says errors_1.txt … errors_5.txt are "Kept in-tree for pattern-matching future failures". They are not in the tree, in this PR or anywhere else.
• §39 and §259 reference a "parent HANDOFF.md" — not in the repo; the only pointer is an absolute path on the author's filesystem.

My suggestion: drop HANDOFF_CCC.md from this PR and fold the still-useful, still-true parts (the layer-by-layer resolution list, the filesystem-layout table with <user> placeholders) into CCC_PODMAN_SETUP.md. CCC_PODMAN_SETUP.md on its own is a good how-to; the handoff mostly duplicates it plus stale state.

---

Security

a. Credential surface documented in HANDOFF_CCC.md §"Credentials". Lines 75-80 publish the internal endpoint https://ete-litellm.ai-models.vpc-int.res.ibm.com, plus the token's length and prefix (<real; 25 chars, prefix "sk-">). The token isn't leaked, but publishing an internal VPC hostname and a token's shape in a public-facing repo is free reconnaissance for no benefit. Please cut the section to "put your endpoint + token in .env; see .env.example".

b. Bearer token on the process command line, on a shared multi-user cluster. run_ccc_smoke.sh:152 (and the doc's smoke recipe) pass --agent-env "ANTHROPIC_AUTH_TOKEN=$ANTHROPIC_AUTH_TOKEN". On CCC that argv is readable by any other user via ps -ef, and it lands in bench.log if bench ever echoes its own invocation. If benchflow supports an env-file or ambient-env passthrough, prefer it; if not, please note the exposure in the doc.

c. curl | sh, unpinned, twice. setup_podman.sh:268 and :271 pipe https://astral.sh/uv/install.sh straight to sh with no version pin and no checksum — baked into a base image every downstream rollout inherits. Slightly awkward, since CCC_PODMAN_SETUP.md §C.1 diagnoses this exact pattern in the SkillsBench verifiers as a failure mode. The doc even shows the verifier pinning uv/0.9.7; the script installs latest, so image contents drift silently between rebuilds. Please pin the version and prefer UV_INSTALL_DIR + a checksum over piping. Same for the doc's docker-compose install (line 104-106): releases/latest/download with no checksum, then chmod +x.

d. Redaction in the env snapshot is allowlist-shaped. sed 's/\(TOKEN\|KEY\)=.*/\1=<REDACTED>/' (run_ccc_experiment.sh:219, run_ccc_smoke.sh:128) catches *TOKEN= / *KEY= and nothing else — a *_SECRET, *_PASSWORD, or Authorization= line goes to disk in cleartext. Invert it: redact everything after = except an explicit safe list.

e. The patched image shadows the official docker.io/library/ubuntu:24.04 tag — please flag the blast radius. setup_podman.sh:275 does podman rmi -f docker.io/library/ubuntu:24.04 and rebuilds under the same tag, with chown/chgrp/useradd/groupadd/usermod/groupmod/adduser/addgroup/dpkg-statoverride all replaced by wrappers that swallow every error (2>/dev/null || :, not just EINVAL), plus TAR_OPTIONS=--no-same-owner in /etc/environment. Consequences worth documenting loudly, or designing away:

• Any other project on that node building FROM ubuntu:24.04 silently gets a mutated base with error-swallowing coreutils and no way to notice.
• Genuine permission bugs inside task images now pass silently, which can turn a real failure into a wrong-but-green verifier result. That matters for benchmark integrity, not just convenience.
• There is no documented un-patch path; the user's pristine cached upstream image is destroyed by the rmi -f.

A distinct tag (localhost/ccc-ubuntu:24.04) plus a documented FROM override would keep the blast radius inside this project. If tag-shadowing is genuinely the only workable route, please say so explicitly and add a "how to revert" section.

---

Accuracy / completeness

f. CCC_PODMAN_SETUP.md §C prescribes an env var that nothing in this repo reads. "the adapter should read SKILLSBENCH_SANDBOX_USER from .env and pass it through" — grep -rn SKILLSBENCH_SANDBOX_USER . finds zero hits outside these new docs, and examples/skillsbench/adapters/adapter.py never passes --sandbox-user (it only sets SANDBOX = "docker" at line 54). So the run_ccc_experiment.sh → cap-evolve → adapter path hits exactly the failure the doc's own troubleshooting table warns about (chown: changing ownership of '/home/agent/.claude': Invalid argument). run_ccc_smoke.sh passes --sandbox-user '' to bench directly and is fine — so the smoke passes and the real run fails, which is the worst arrangement. Either land the adapter change alongside, or mark §C as "requires an adapter change not yet in main" and say so in run_ccc_experiment.sh.

g. CCC_PODMAN_SETUP.md §"Batch (LSF) mode — TODO" is stale. It ends with "Draft plan (untested) … Detailed instructions will follow after we've done a batch dry-run" — while this same PR adds submit_ccc_experiment.sh, which implements precisely that plan. Replace the TODO with a pointer to the script.

h. Section ordering. The setup doc runs §A → §B → §C.1 → §C. C.1 before C reads like a mis-merge; renumber.

i. Portability of the docker-shim assertion. [[ "$(readlink -f "$(which docker)")" != "$HOME/.local/bin/docker" ]] (run_ccc_experiment.sh:192, run_ccc_smoke.sh:102, and the doc's sanity check at line 619) compares a fully-resolved path against an unresolved $HOME. If any component of $HOME is a symlink — common when /u/<user> fronts GPFS — this trips and run_ccc_experiment.sh aborts with exit 3 on a correctly configured node. Compare readlink -f on both sides.

j. Smoke task disagreement. run_ccc_smoke.sh defaults to invoice-fraud-detection ("the one we know passes with seed skills"); both docs use offer-letter-generator as the smoke task, and §C.1 reports it as the one confirmed fixed by the v7 image. Pick one, or explain why they differ.

k. Small script nits.

• run_ccc_smoke.sh header (line 30) documents a marker file named PASS/FAIL/ERROR; the code writes a file named OUTCOME containing that word (lines 105/203).
• run_ccc_smoke.sh:161-162 — comment says "Newest run subdir (timestamped)" but find … | head -1 returns filesystem order, not newest. Harmless today because rm -rf "$JOBS" precedes it, but the comment will mislead the next person, and it silently picks one of N trials.
• --help output is truncated mid-section by hardcoded sed ranges: sed -n '2,25p' (smoke, cuts the output-layout list) and sed -n '2,40p' (submit).
• submit_ccc_experiment.sh header example (line 16) pairs --queue x86_1h with --walltime 6:00.
• submit_ccc_experiment.sh:97 passes -M 64G. LSF -M units depend on LSF_UNIT_FOR_LIMITS; can you confirm CCC's LSF accepts the G suffix rather than needing -R "rusage[mem=64G]"?
• setup_podman.sh:245 export PYTHONPATH=… (in run_ccc_experiment.sh) clobbers any inherited PYTHONPATH instead of prepending.
• setup_podman.sh leaks STORAGE_BASE, RUN_BASE, CONF_DIR, CONF_FILE into the sourcing shell (the unset at line 297 misses them) — generic names in a shell the user keeps using.
• setup_podman.sh:70-71 overwrites an existing ~/.config/containers/storage.conf with no backup. It's described as idempotent, but it's destructive to a hand-maintained file. A .bak on first write would be cheap.
• setup_podman.sh has a #!/bin/bash shebang, is chmod +x, and is meant to be sourced; line 98's ${var//…} is bash-only. Worth a one-line "must be sourced from bash" note.

---

Placement / conventions

l. third_party/skillbench/docs/SKILLSBENCH_INVENTORY.md is in the wrong place. Two problems:

• third_party/ in this repo holds vendored upstream code with provenance — see third_party/spreadsheetbench/ with its NOTICE.md + README.md. SKILLSBENCH_INVENTORY.md is our own analysis (it contains our baseline numbers, "our current 4-skill seed package", and a cap-evolve-vs-EvoSkills strategy section). It isn't third-party content and carries no upstream notice.
• The directory is spelled skillbench while the benchmark is skillsbench everywhere else in the tree: examples/skillsbench/, ci/benchmarks/skillsbench/, docs/REPRODUCE_skillsbench.md.

docs/evidence/ or examples/skillsbench/ would fit the existing structure better.

m. Nothing links to the new docs. README.md:304 is the single entry point for how-to guides and points only at cap-evolve-with-exgentic-tau2.md. Please add a row for the CCC guide, otherwise these 900 lines are unreachable except by find.

n. Generalizability — clean, and I want to note it. I checked the diff for code changes: there are none outside scripts/ccc/. Nothing under core/ or skills/ is touched, so no CCC- or SkillsBench-specific assumption leaks into the algorithm surface. The SkillsBench-specific values in run_ccc_smoke.sh (claude-agent-acp, claude-opus-4-6, the task id, seed_capability) are fine where they are — a CCC smoke script is legitimately environment-specific, and an environment-specific setup guide is not a generalizability violation. No objection on that axis.

---

Verification I ran

• bash -n on all four scripts: clean.
• submit_ccc_experiment.sh --dry-run: fails as described in (3).
• run_ccc_experiment.sh --dry-run: reaches the spec check and exits cleanly with a good error message.
• Confirmed cap-evolve run accepts every flag the scripts pass (--spec --project --run-ts --resume --max-iterations), and that --spec .capevolve/project/capevolve.yaml with --project .capevolve/project resolves correctly after the cd "$PROJECT_ROOT" — _resolve_spec resolves an explicit --spec against the cwd and its inside-project check passes. No bug there, though --spec is redundant since it's the default.
• Core suite green on this branch: 954 passed, 12 skipped. Nothing in shared config was disturbed. (Note the branch is 9 commits behind origin/main; a rebase before merge would be good hygiene.)

---

Recommendation

Request changes. Must-fix before merge: (1) source the in-repo setup_podman.sh, (2) de-hardcode CE_BIN, (3) fix the --dry-run mkdir and CCC_LOGS default, (4) green CI (DCO sign-off + the two dangling links), (5) /results/ in .gitignore. I'd also drop HANDOFF_CCC.md, trim the credentials section, move the inventory out of third_party/, and pin the uv install.

Once the scripts run from a clean checkout and CI is green, this is a solid contribution — the setup guide is the kind of institutional knowledge that usually evaporates, and it's worth keeping.

— Reviewed by Osher-Elhadad

[bcarmeli]

Thanks — this was a genuinely useful review; the account-portability problems were real and the redaction one was worse than reported. All blocking items are fixed and CI is green (DCO included). Pushed as three commits, then rebased with --signoff so all five carry the trailer.

Blocking

1. In-repo setup_podman.sh — both runners now source "$SCRIPT_DIR/setup_podman.sh", overridable via $CCC_SETUP_PODMAN, erroring clearly if unreadable. You were right that the docs half mattered more: eight references told readers to source the external path. All repointed.
2. CE_BIN — $CE_BIN → $PROJECT_ROOT/.venv → $PATH, with an error naming all three. Also documents that a git worktree doesn't inherit the main checkout's .venv, which is how we actually hit this.
3. --dry-run — mkdir moved after the guard; CCC_LOGS defaults to $PROJECT_ROOT/results/.ccc_logs. Verified with the exact invocation from the script header: rc=0, nothing written.
4. CI — both dangling links gone, all commits signed off.
5. /results/ in .gitignore.

Security

(d) was the important one. The sed only caught *TOKEN=/*KEY=, so I tested the old expression against a .env containing DB_PASSWORD, MY_SECRET and Authorization= — all three were being written to env_snapshot.txt in cleartext. Now an allowlist of known-non-secret keys, everything else redacted, verified against that same file.

• (c) uv pinned via ARG UV_VERSION=0.9.7, matching the version §C.1 shows the verifiers pinning.
• (e) New "Blast radius" section: the wrappers swallow every error, so a real permission failure can score as a pass — called out explicitly as a benchmark-integrity risk, not just convenience. Added the revert recipe, which didn't exist. Kept tag-shadowing with the reason stated (task images have FROM ubuntu:24.04 baked in and we can't rewrite upstream Dockerfiles), plus a note that a private tag is the right fix if benchflow gains a base-image override.
• (a) Credentials section cut to "put it in .env".
• (b) Documented the argv exposure at the call site.

Accuracy

• (f) Not papered over — flagged in both the doc and run_ccc_experiment.sh, since "smoke green, real run fails" is the trap worth naming loudly.
• (i) Both sides now readlink -f'd.
• (g) Replaced with the tested procedure. The draft plan's wall-time advice turned out to be actively harmful: these jobs finish and then hang, so -W killed them after the result was written (TERM_RUNLIMIT), which cost us real runs. So I went further than a pointer and fixed the wrapper — no -W unless explicitly asked, -n 1 for --max-iterations 0, and a new --host for the one-host-per-job pin it had no flag for. Also documents that cccxc600–630 is permanently reserved for another group while showing as ok/idle in bhosts — bhosts doesn't show reservations, only brsvs does.
• (l) Inventory deleted, not moved: a byte-identical copy already exists on the skillbench-history branch at insights/SKILLSBENCH_INVENTORY.md.
• (m) README how-to row added.
• (k) All of them: OUTCOME naming, the misleading "newest run subdir" comment, both truncated --help ranges, PYTHONPATH prepend, the four leaked vars, storage.conf .bak, the "must be sourced from bash" note, and the stale --queue/--walltime header pairing. On -M units — confirmed empirically: CCC's LSF accepts 64G and reports MEMLIMIT back as 64 G, so no rusage[mem=] needed. Documented in the script.

HANDOFF_CCC.md — dropped, per your recommendation

Agreed, and the deciding factor was your contradiction point. It claimed the smoke passed while Where we paused said errors=1 and named aardvark-dns as the live blocker — but that was resolved, and the scripts in this PR have since run an 87-task benchmark and an 8-fold transfer pilot. It documented a state that no longer exists.

Rather than relocate a self-contradictory doc, it's deleted; git history keeps it at git show 1bf71670:docs/how-to/ccc/HANDOFF_CCC.md. Four durable pieces moved into CCC_PODMAN_SETUP.md:

• "Layer index: symptom → fix" — your favourite part of the PR was the troubleshooting table, and the handoff's 13-layer list was the missing scannable index to it. Now a table cross-referenced to the detail sections.
• "What this touches outside the repo" — every path the setup writes, so readers know their own blast radius, including that uv tool install --force benchflow wipes the §A/§B patches.
• "If you ever get admin help, delete all of it" — all of this routes around one missing subuid range; worth saying plainly.
• The Modal/benchflow-0.6.5 limitation (already under "What's NOT solved").

Not addressed

(h) §C.1-before-§C ordering. A real mis-merge, but reordering that much prose felt better done deliberately than folded in here — happy to do it in this PR if you'd rather it not linger. Also still 9 commits behind main; will rebase before merge as you suggested.