Build time experiment images - #686
Closed
RileyFW wants to merge 2 commits into
Closed
Conversation
Runner security hardening: - runner.py: run user-submitted trials with a sanitized environment (_build_trial_env allowlist) so untrusted code can no longer read the runner's injected secrets (GMAIL_CREDS, MONGODB_PORT, BACKEND_PORT) via os.environ. - job-runner.yaml: add a securityContext (allowPrivilegeEscalation:false, drop ALL capabilities, RuntimeDefault seccomp) and CPU/memory requests+limits so one experiment cannot starve its node. Build-time per-experiment images (Phases 0-2): - Phase 0: runner-base.Dockerfile (interpreter + harness base image); kubernetes_init/registry/ documents the in-cluster ctlptl registry; Tiltfile builds runner-base (consumed via the backend RUNNER_BASE_IMAGE env). - Phase 2: build_image.py + job-builder.yaml render a Dockerfile from the declared deps, build a per-experiment image FROM runner-base via an in-cluster Kaniko Job (content-hash cached in the registry), and return the ref. app.py spawns the build off the request thread then runs the runner Job from the built image; spawn_runner.create_job_object takes an image_override. Returns None when no deps are declared, so current experiments are unchanged. - Phase 1: frontend declares pipRequirements / aptPackages (db_types.ts, InformationStep.tsx, NewExperiment.tsx); experiment.py mirrors the fields; app.py reads them from Mongo before building. Supporting changes: - RBAC (tilt + backend cluster roles): grant jobs get/list/watch and configmaps create/delete/get for the build orchestration. - deployment-backend.yaml: REGISTRY_HOST / RUNNER_BASE_IMAGE env. - app.py: log background spawn_job failures via a Future done-callback instead of silently swallowing them. Verified end-to-end via tilt ci in dev/minikube: stack green, Kaniko build FROM runner-base pushes glados-exp:<hash>, runner Job spawned from it, build cache hit on repeat. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 3 of the build-time per-experiment images work. With dependencies now baked into the image at build time, the runner no longer needs to install anything at runtime, so it can run fully locked down: - job-runner.yaml: run as non-root (uid/gid 1000, fsGroup 1000), readOnlyRootFilesystem, drop ALL capabilities, seccomp RuntimeDefault. Writes go to emptyDir volumes at /work (workingDir) and /tmp; HOME=/tmp. Resources set a memory limit (node protection via OOM-kill) and CPU request, but no CPU limit -- a hard CPU limit is enforced via the cgroup CFS quota, which the WSL2 dev kernel rejects with EINVAL, crashing the pod at container init. Clusters with CFS-quota support can re-add it. - spawn_runner.py: reference the runner by its absolute /app/runner.py path, since workingDir is now the writable /work volume. - runner.py: remove the runtime apt-get / pip install / arbitrary admin-command install paths; dependencies are baked at build time. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.