Run the launchers in a long-lived pod on Kubernetes/OpenShift (Apptainer runtime) - #80
Merged
Merged
Conversation
Claude-Session: https://claude.ai/code/session_01Bsm8NJasSPeyJ9xeamuuch --- Signed-off-by: Guillaume Moutier <guimou@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com>
…redentials on PVC, repos layout Claude-Session: https://claude.ai/code/session_01Bsm8NJasSPeyJ9xeamuuch --- Signed-off-by: Guillaume Moutier <guimou@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com>
Runs every launcher against stub podman/gh/npm in a fixed environment and compares the rendered 'podman run' argv with tests/golden. Recorded on the unmodified engine so the runtime-backend refactor can prove it changes nothing. Claude-Session: https://claude.ai/code/session_01Bsm8NJasSPeyJ9xeamuuch --- Signed-off-by: Guillaume Moutier <guimou@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com>
Wrappers and the engine no longer append raw `podman run` flags. Mounts and environment variables are declared with add_mount / add_optional_mount / add_env into an ordered BOX_SPEC, and a runtime backend renders that spec, the image tag and the launch command into the final command line. The Podman backend is the only one and keeps today's behavior exactly (rootless flags, :z labels, firewall capabilities, macOS handling); a `nolabel` mount option marks the system paths that were never relabeled (/etc/localtime, X11 and Wayland sockets). No behavior change: the golden test (recorded on the previous engine, including the new scenarios) passes unchanged for all four launchers. An old wrapper paired with this engine is detected and refused instead of silently launching without its mounts. Also: tests.yml CI workflow (shellcheck + golden test), docs updated (development, architecture, contributing, AGENTS.md), plan updated to the single-file layout. Claude-Session: https://claude.ai/code/session_01Bsm8NJasSPeyJ9xeamuuch --- Signed-off-by: Guillaume Moutier <guimou@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com>
Second backend for the pod-as-host case. Selected with --runtime, else CODEBOX_RUNTIME, else podman when installed, else apptainer. The registry image is converted once to $CODEBOX_SIF_DIR/<box>-<tag>.sif (temp name + atomic move; --pull re-converts, --local refuses to convert) and run with `apptainer exec --userns --no-home --no-mount tmp,cwd --pid --ipc --cleanenv --writable-tmpfs --pwd /workspace` plus the session spec as --bind flags. Environment goes through APPTAINERENV_* exports (survive --cleanenv, not split on commas like --env). Optional CODEBOX_SCRATCH_DIR binds a per-session directory at /tmp. Sessions are tracked with marker files (host, PID, start time) since Apptainer rewrites its command line. Runtime capabilities gate host features: clipboard, audio and npm-global are skipped, --with-firewall and --build are refused with a pointer to the pod-level alternative. The Podman path is unchanged (golden files untouched); ten apptainer scenarios added, three of them refusals. Docs: usage.md Runtimes section (flags, env vars), architecture.md, AGENTS.md, plan updated. Claude-Session: https://claude.ai/code/session_01Bsm8NJasSPeyJ9xeamuuch --- Signed-off-by: Guillaume Moutier <guimou@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com>
Run the launchers inside a long-lived pod on Kubernetes/OpenShift, the pod playing the role of the workstation: - k8s/Containerfile: slim UBI9 image (quay.io/guimou/codebox-pod) with rootless Apptainer + FUSE helpers, tmux, git/gh, and the launchers in the flat layout. apptainer.conf gets a 4 GB session dir and no default home/tmp mounts. Runs as coder/1000 with CODEBOX_RUNTIME=apptainer and the scratch/cache dirs on node-local /scratch. - k8s/entrypoint.sh: home skeleton on the RWX PVC (~/repos, SIF store, session markers, .bashrc), then idles; sessions run from tmux. - k8s/cluster: SCC codebox-apptainer (restricted-v2 + seccomp unconfined, MustRunAsNonRoot, fsGroup RunAsAny) and a ClusterRole to use it. - k8s/base + overlays/example: service account without token, RoleBinding, RWX PVC mounted at /home/coder (storage class set by the overlay), Deployment with the /dev/fuse annotation, UID/fsGroup 1000, Recreate strategy, optional codebox-env Secret, node-local emptyDir at /scratch. - k8s/gen-egress-firewall.sh: OVN EgressFirewall from the firewall domain lists, the pod-level replacement for --with-firewall. - build-pod.yml workflow; tests.yml also renders the kustomizations and runs the generator; docs/kubernetes.md (deploy, daily use, PVC layout, security posture, troubleshooting); README, AGENTS.md, development.md, CONTRIBUTING.md and the plan updated. Real env files are git-ignored. Nothing here has run on a cluster yet; the gate list in the plan is next. Claude-Session: https://claude.ai/code/session_01Bsm8NJasSPeyJ9xeamuuch --- Signed-off-by: Guillaume Moutier <guimou@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bsm8NJasSPeyJ9xeamuuch --- Signed-off-by: Guillaume Moutier <guimou@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bsm8NJasSPeyJ9xeamuuch --- Signed-off-by: Guillaume Moutier <guimou@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Same launchers, same per-project isolation, but with a long-lived pod playing the role of the workstation:
oc rshin,cd ~/repos/<repo>, runccbox/ocbox/qcbox/cxbox. The harness runs through Apptainer from a SIF converted from the regularquay.io/guimou/<box>image. Design and decisions indocs/plans/kubernetes-apptainer.md; user guide indocs/kubernetes.md.Commits, in order:
docs/plans/kubernetes-apptainer.md) with the decisions: extend this repo rather than a new project, storage class as a parameter (any RWX class), UBI9 pod image, credentials on the PVC as on a host, repos under~/repos/<name>.tests/render-test.sh,tests/golden/): every launcher against stubpodman/gh/npmin a throwaway home; the rendered command line is compared with recorded expectations. Recorded on the unmodified engine.add_mount/add_optional_mount/add_envinto an ordered session spec; a runtime backend renders it. The Podman backend keeps today's flags exactly (nolabelmarks the system paths that were never relabeled). All golden files unchanged. An old wrapper with the new engine is refused instead of launching without mounts.--runtime/CODEBOX_RUNTIMEselection, SIF conversion with atomic move (--pullto refresh,--localto refuse),apptainer exec --userns --no-home --no-mount tmp,cwd --pid --ipc --cleanenv --writable-tmpfs --pwd /workspace+ explicit binds, env viaAPPTAINERENV_*exports (--envsplits on commas), optional per-session/tmponCODEBOX_SCRATCH_DIR, session marker files for--list-sessions. Clipboard/audio/npm-global skipped,--with-firewalland--buildrefused with pointers. Ten new scenarios, three of them refusals.k8s/. UBI9 pod image (quay.io/guimou/codebox-pod: Apptainer, tmux, gh, launchers; 4 GB session dir), entrypoint, SCC + ClusterRole (k8s/cluster, cluster-admin once), kustomize base + example overlay (token-less SA, RWX PVC over/home/coder, Deployment with/dev/fuseannotation, UID/fsGroup 1000, node-local/scratch),gen-egress-firewall.sh(OVN EgressFirewall from the firewall domain lists),build-pod.yml, docs.Verification
shellcheckon launchers, engine, test, and the twok8s/scripts.tests/render-test.sh: 25 scenarios pass (15 Podman, recorded before the refactor; 10 Apptainer).kubectl kustomize k8s/overlays/exampleandk8s/clusterrender; the egress generator emits one rule per domain.tests.ymlworkflow runs all of the above on PRs.Not verified
Nothing has run on a cluster or a real Apptainer yet. The gate list in the plan is the next step; the points most likely to need a follow-up are the image's
/home/coderunder--no-home+ writable overlay, credential file binds on top of directory binds,sessiondir max size, andfsGroupon the target RWX class (EFS needs an access point with uid/gid 1000).Note: merging triggers
release.yml(it rebuilds all harnesses onlib/changes) and the newbuild-pod.yml.https://claude.ai/code/session_01Bsm8NJasSPeyJ9xeamuuch