docs(ccc): batch-submission scripts + rootless-podman recipe for CCC

[bcarmeli]

Adds a reproducible userspace path to run cap-evolve on IBM CCC clusters (rootless podman, no admin, no /etc/subuid entry, no systemd on compute nodes). Nine layered workarounds cover image unpack, apt sandboxing, package-postinst chowns/useradds/dpkg-statoverride, private dbus, network_mode=host (aardvark-dns bypass), docker-shim PATH ordering, and docker-compose cp UID mismatches. Ships with a smoke script, a full experiment script (with --resume support for LSF walltime), an LSF submitter, and a colleague-facing README.

• docs/RUN_ON_CCC.md — walkthrough + troubleshooting keyed to each workaround layer + sanity-check script
• scripts/ccc/setup_podman.sh — one-shot podman/dbus/socket setup, builds a patched ubuntu:24.04 with ownership-wrapper shell scripts
• scripts/ccc/run_ccc_smoke.sh — single-task LSF-ready smoke
• scripts/ccc/run_ccc_experiment.sh — full cap-evolve run wrapper, supports --resume for walltime recovery
• scripts/ccc/submit_ccc_experiment.sh — LSF bsub wrapper with sensible defaults for baseline vs full-iter jobs

What & why

Checklist

• python -m pytest core/tests -q passes
• python skills/_registry/build_manifest.py skills is clean (no errors)
• Any new/changed skill's scripts/check.py is green
• Honesty invariants untouched (gate on val, test sealed) — or explained
• Docs updated (README/skill SKILL.md) if behavior changed

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[skillberry-bot]

🏷️ Automatic Labeling

I've analyzed this pull request and added the following labels:

• documentation - dx - documentation - dx - enhancement

These labels were selected based on the PR title, description, and changed files. If you believe any labels are incorrect or missing, feel free to adjust them manually.

[OsherElhadad]

Review — #243 (rootless-podman recipe + CCC batch scripts)

Checked out add-ccc-support (765e425), fetched origin/main (49fcedb), read all 5 added files in full, syntax-checked and dry-ran the scripts, ran the core suite, and diffed this branch against #412 file-by-file. Verdict up front, because it determines everything else.

---

The #412 question: #243 is superseded — but #412 regressed three things #243 got right

This is not "similar ground." It is the same work, and #412 is the later revision of it. Concretely:

Path in #243	Counterpart in #412	Delta (git diff 243..412)
scripts/ccc/setup_podman.sh	same path	+65 / −4 (v5 → v7 image: TAR_OPTIONS, uv/uvx preinstall, ca-certificates)
scripts/ccc/run_ccc_experiment.sh	same path	+4 / −10
scripts/ccc/run_ccc_smoke.sh	same path	+3 / −3
scripts/ccc/submit_ccc_experiment.sh	same path	+1 / −1
docs/RUN_ON_CCC.md (604 lines)	docs/how-to/ccc/CCC_PODMAN_SETUP.md	+57 / −6 — same document, relocated, with the v6/v7 notes appended

So #412 is a strict content superset: every file here appears there, evolved forward, plus HANDOFF_CCC.md and the SkillsBench inventory. #412 also places the doc correctly under docs/how-to/ccc/ — this PR's docs/RUN_ON_CCC.md sits at the top level of docs/, where nothing else host-specific lives and where nothing links to it (README.md's how-to row points only at cap-evolve-with-exgentic-tau2.md; grep -rn RUN_ON_CCC across README.md/CHANGELOG.md/llms.txt finds zero references — these 604 lines are reachable only by find).

Merging both would be a straight collision — four identical paths, two divergent copies of the same 600-line doc.

But — and this is the part that matters for the merge decision — #412 is worse than #243 in three places, and the fix is to port #243's version forward:

1. run_ccc_experiment.sh CE_BIN. #243 (lines 248-258) resolves the CLI properly:

   CE_BIN="${CAP_EVOLVE_BIN:-$(command -v cap-evolve || true)}"

   with a helpful two-line error. #412 replaced this with a hardcoded
   /dccstor/knewedge2/boazc/workarea/python/skillberry_ai/cap-evolve/.venv/bin/cap-evolve.
   #243's version is the correct one and should survive.

2. source of setup_podman.sh. #243 uses source "$SCRIPT_DIR/setup_podman.sh" (experiment:179, smoke:90) — resolves from the checkout, works for anyone. #412 replaced both with source /dccstor/knewedge2/boazc/workarea/python/setup_podman.sh, i.e. the in-repo script it ships is never the one that runs. #243 is right.

3. CCC_LOGS default. #243: ${CCC_LOGS:-$HOME/ccc_logs}. #412: ${CCC_LOGS:-/dccstor/knewedge2/boazc/ccc_logs}. Same story — and #243's setup_podman.sh:42 keeps the GPFS example as a commented /dccstor/<your-project>/<your-user>/ placeholder, where #412 substituted the real personal path.

Net: close #243 in favour of #412, and in #412 revert those three hardcodings to exactly what this PR has. I've noted the same three as must-fixes on #412. The setup_podman.sh v6/v7 content, the doc placement, and ca-certificates should come from #412; the path handling should come from here.

---

Verification I ran on this branch

• bash -n on all four scripts: clean.
• submit_ccc_experiment.sh --suite-id test --max-iterations 0 --dry-run: works, prints a well-formed bsub line and exits 0. (Note it mkdir -p "$CCC_LOGS" at line 90 before the dry-run check at 123, so a dry run creates ~/ccc_logs as a side effect — move the mkdir below the DRY_RUN guard. Minor, but a dry run should touch nothing. This bug is still present in #412.)
• run_ccc_experiment.sh --dry-run: reaches the spec check and exits 2 with a good actionable message.
• Core suite on this branch: 179 passed (.venv-tau2/bin/python -m pytest core/tests -q). Nothing broke — expected, since no Python is touched.
• git merge-tree HEAD origin/main: merges clean, no conflicts, despite being 148 commits behind origin/main. The tree drift isn't a blocker because the PR only adds new paths.
• CI: green apart from one CANCELLED "aggregate history". No Signed-off-by trailer on 765e425 — DCO will need a sign-off if this were to proceed.

Generalizability — clean, and worth recording

No code outside scripts/ccc/ and one new doc. Nothing under core/, skills/, or any shared config path. The SkillsBench-specific values in run_ccc_smoke.sh (claude-agent-acp, claude-opus-4-6, invoice-fraud-detection, seed_capability) are fine where they are: a CCC smoke script is legitimately environment-specific and a host setup guide is not a generalizability violation. No objection on that axis.

Security notes (all also apply to #412)

• Env-snapshot redaction is allowlist-shaped. sed 's/\(TOKEN\|KEY\)=.*/\1=<REDACTED>/' (run_ccc_experiment.sh:219, run_ccc_smoke.sh:128) catches *TOKEN=/*KEY= only. I ran it against a synthetic .env: CLIENT_SECRET=, DB_PASSWORD=, and an Authorization= line all pass through in cleartext into env_snapshot.txt. Invert the logic — redact everything after = except an explicit safe list.
• results/ is not in .gitignore. git check-ignore results/x/y/env_snapshot.txt → not ignored. These scripts write .env snapshots and full run logs under $PROJECT_ROOT/results/, so a stray git add -A on a compute node commits credentials-adjacent material. .gitignore already covers .capevolve/ and .bench_runs/; add /results/.
• Bearer token on the argv, on a shared cluster. run_ccc_smoke.sh:152 passes --agent-env "ANTHROPIC_AUTH_TOKEN=$ANTHROPIC_AUTH_TOKEN". On CCC that command line is readable by any other user via ps -ef. Prefer an env-file passthrough if benchflow has one; otherwise document the exposure.
• Unpinned download-and-run in the doc. Line 94-96 fetches docker-compose from releases/latest/download with no checksum, then chmod +x. Pin a version and verify a digest. (#412 adds two curl https://astral.sh/uv/install.sh | sh calls inside the base image with the same problem, so this gets worse there, not better.)
• The patched image shadows the official ubuntu:24.04 tag. setup_podman.sh:214 does podman rmi -f docker.io/library/ubuntu:24.04 and rebuilds under the same tag, with chown/chgrp/useradd/groupadd/usermod/groupmod/adduser/addgroup/dpkg-statoverride all replaced by wrappers that swallow every error (2>/dev/null || :, not just EINVAL). Two consequences the doc should state loudly: any other project on that node building FROM ubuntu:24.04 silently inherits error-swallowing coreutils, and genuine permission failures inside task images now pass silently — which can turn a real failure into a wrongly-green verifier result. That's a benchmark-integrity concern, not just a convenience one. A distinct tag (localhost/ccc-ubuntu:24.04) plus a documented FROM override would confine the blast radius; the user's cached upstream image is destroyed by the rmi -f with no documented revert.

Two accuracy problems specific to this PR

• The doc's final section contradicts the PR. docs/RUN_ON_CCC.md:586 — "## Batch (LSF) mode — TODO … Draft plan (untested) … Detailed instructions will follow after we've done a batch dry-run" — while this same PR adds submit_ccc_experiment.sh, which implements exactly that four-step plan. The PR title leads with "batch-submission scripts" and the doc then tells the reader batch mode isn't done. (Still stale in #412 too.)
• The doc never mentions its own scripts. grep -n 'run_ccc_experiment\|submit_ccc\|run_ccc_smoke' docs/RUN_ON_CCC.md → zero hits. The four scripts ship undocumented; the doc's verification section hand-rolls a bench eval run instead of pointing at run_ccc_smoke.sh.
• Smoke-task disagreement: the script defaults to invoice-fraud-detection ("the one we know passes with seed skills"), the doc's smoke recipe (line 487) uses offer-letter-generator.
• run_ccc_smoke.sh header line 30 documents a marker file named PASS/FAIL/ERROR; the code writes a file named OUTCOME containing that word (lines 105, 203).
• The docker-shim assertion (experiment:192, smoke:102) compares readlink -f "$(which docker)" against an unresolved $HOME/.local/bin/docker. On CCC, /u/<user> commonly fronts GPFS via a symlink — in which case this trips and the script aborts exit 3 on a correctly configured node. readlink -f both sides.

---

Recommendation

Close #243 as superseded by #412, rather than fixing it in place — #412 carries the same five artifacts forward with the v7 image work and the correct docs/how-to/ccc/ placement, and maintaining two copies of a 600-line doc is worse than either.

Condition on closing: #412 must first adopt this PR's generic path handling — the CAP_EVOLVE_BIN/command -v resolution, source "$SCRIPT_DIR/setup_podman.sh", and ${CCC_LOGS:-$HOME/ccc_logs}. Those three are strictly better here than there, and closing #243 without porting them would lose real work and leave #412 unrunnable outside one person's home directory. I've flagged them as must-fixes on #412 as well, so the ordering is: land the corrected #412, then close this.

Genuine credit where it's due: the workaround archaeology in this document — nine layered fixes, each with the specific failure it cured and a troubleshooting table keyed back to it — is exactly the institutional knowledge that normally evaporates into someone's shell history. It's worth keeping. It just belongs in one place, at the newer revision.

— Reviewed by Osher-Elhadad

[bcarmeli]

Superseded by #412 (merged), which is a strict superset of this PR's files.

Per @OsherElhadad's review here, #412 originally regressed three things this PR got right — account-portable CE_BIN resolution, sourcing setup_podman.sh from a repo-relative path, and a portable CCC_LOGS default instead of a hardcoded personal path. All three were folded into #412 before it merged, so the closing condition from that review is satisfied. Closing this one in favor of #412.