diff --git a/.commitrail/changes/pilot-gvisor-offline-build-spike.md b/.commitrail/changes/pilot-gvisor-offline-build-spike.md new file mode 100644 index 000000000..bc59e0e74 --- /dev/null +++ b/.commitrail/changes/pilot-gvisor-offline-build-spike.md @@ -0,0 +1,83 @@ +# PILOT-00 — Establish The Offline gVisor Build Boundary + +- Initiative: None +- Durable disposition: Complete +- Intended merge outcome: Preserve a reproducible feasibility result that either identifies a bounded offline image-build and oracle-test mechanism under gVisor or gives PILOT-04 and PILOT-06 concrete evidence to select a separate build boundary. + +## Intent + +Resolve the highest-risk execution unknown before the external-checker launcher is designed: whether a contributor Dockerfile and its oracle can be built and exercised inside gVisor with networking disabled, without a host container socket or privileged sandbox. The spike must distinguish an unsupported infrastructure path from a contributor work failure and must not create a product runtime. + +## Current behavior + +Current `main` provides the local API/worker/PostgreSQL/Redis/MinIO stack but no external-checker launcher or gVisor runtime. [`docs/roadmap_status.md`](../../docs/roadmap_status.md) records the external runner as remaining pilot work. PILOT-04 and PILOT-06 require the PILOT-00 result before selecting their build and execution design. + +The Linux probe established that the maintained Kaniko fork can build the included digest-pinned Alpine fixture inside gVisor from a sealed cache. A second gVisor sandbox ran its oracle offline. Rootless BuildKit could not establish its subordinate user mapping on this host under either gVisor or an ordinary `runc` control. The host still has no registered `runsc` runtime; an isolated trusted Docker-in-Docker harness supplied the runtime registration for the experiment only. + +## Bounded change + +### Allowed + +- `.commitrail/changes/pilot-gvisor-offline-build-spike.md` for this bounded intent, evidence and disposition. +- `experiments/pilot00_gvisor_offline/**` for a minimal sample Dockerfile, oracle, pinned-input manifest and reproducible probe. +- `docs/engineering/pilot00-gvisor-offline-build-spike.md` for the measured decision input and explicit platform limits. +- Narrow PILOT-00/PILOT-04/PILOT-06 facts in `docs/roadmap_status.md` after the evidence changes the current pilot dependency. +- GitHub issue references that link the final recommendation from #491 and #493 after review. + +### Not allowed + +- Backend, frontend, CLI, database schema/migration, authorization, policy, catalogue, composition-root or production workflow changes. +- A product launcher, checker SDK, model proxy, hosted deployment, Kubernetes path or macOS implementation. +- A privileged submitted-code sandbox, host Docker/containerd socket in the sandbox, writable host cache, enabled external network during build/test, or secret injection. +- Removal or mutation of existing containers, images, volumes, networks, local pilot data or unrelated host configuration. +- Treating Docker Desktop's ordinary Linux runtime as gVisor or claiming hosted/macOS evidence that was not executed. + +## Design and decisions + +Use a project-scoped disposable probe with exact tool/base identities and retained machine-readable measurements. The selected builder is a non-privileged Kaniko container running as root inside gVisor with only `CHOWN`, `DAC_OVERRIDE`, `FOWNER` and `SYS_CHROOT`; it has external networking disabled, no host socket, no `SYS_ADMIN` and read-only digest-bound offline inputs. Run the oracle against the produced image in a separately identified, non-root and capability-free gVisor execution boundary. + +Installing or registering a runtime on the development host is probe setup, not product code. This host was not mutated: a privileged, networkless outer Docker-in-Docker container acted as trusted development infrastructure and registered `runsc` only for its inner daemon. This does not prove a privilege-free deployment. A failed mechanism is a valid result when the failure is reproduced and classified without silently relaxing isolation. + +## Acceptance criteria + +- [x] Record the exact Linux host, Docker, gVisor, builder and pinned base-image identities used by the probe. +- [x] Build one sample Dockerfile and run its oracle under gVisor with external networking disabled, no host socket and no privileged sandbox, or preserve a reproducible reason the mechanism cannot satisfy that boundary. +- [x] Prove the offline cache/base identity, sandbox user/capability/socket/network facts and output identity without trusting submitted output alone. +- [x] Measure elapsed time, sampled CPU, peak observed memory and disk use for the stages that this host can honestly execute. +- [x] Demonstrate a genuine Dockerfile/work failure separately from runtime, cache, builder and limit failures. +- [x] Record a provisional tested envelope and the effect of timeout, memory and disk limits without converting infrastructure failure into contributor failure. +- [x] State the Docker Desktop/macOS development fallback and every unverified platform limit accurately. +- [x] Give PILOT-04 and PILOT-06 an explicit recommendation and link the reviewed result from [#491](https://github.com/Flow-Research/workstream/issues/491#issuecomment-6059462336) and [#493](https://github.com/Flow-Research/workstream/issues/493#issuecomment-6059463258). + +## Risk and review routing + +- Risk class: L1 +- Required reviewers: architecture, security, qa, test_delta, documentation, product_ops +- Human review focus: Whether the executed evidence proves the named isolation boundary, whether any workaround weakens it, and whether the recommendation should keep image builds inside gVisor or move them to a dedicated build VM boundary. + +## Evidence + +| Claim | Command or proof | Result | Remaining uncertainty | +|---|---|---|---| +| Current host and runtime boundary | `uname`, Docker inventory, pinned `runsc` archive checksum, outer/inner container inspections | Ubuntu 24.04 / Linux 6.8 x86_64, Docker 29.1.3; host daemon retains only `runc`; inner daemon registers `runsc release-20260928.0` | Hosted execution-node registration and hardening remain unproved | +| Offline build and oracle boundary | `PILOT00_RUN_ID=replay1 PILOT00_STATE_DIR=/tmp/ws-pilot00-replay1 PILOT00_EVIDENCE_ID=20261008T120000Z ./experiments/pilot00_gvisor_offline/probe.sh run` | PASS: 16.420 s build and 3.226 s second-sandbox oracle; result JSON SHA-256 `afc06f12ed50606372147a97824467ec448d28037ab8a5baa8dc26616b844c81` | Tiny Alpine fixture only; output is attempt-bound but not byte-reproducible across builds | +| Isolation and resources | Builder/oracle launch inspections, in-sandbox assertions and sampled stats under `/tmp/ws-pilot00-replay1/results/20261008T120000Z/` | `runsc`, network none, no socket, builder non-privileged with four bounded in-sandbox capabilities; oracle UID 65532/capability-free; one-CPU build peak sample 27,283,948 bytes and 35 PIDs | Trusted outer DIND is privileged development infrastructure; representative sizing is unproved | +| Failure classification | Intentional exit, empty cache, forced stop, 16 MiB and 1 MiB output probes in the same replay | Exit 42 stayed a work failure; cache/deadline/OOM/disk cases remained infrastructure candidates | PILOT-04 owns final transport-neutral mapping and retry policy | +| Rootless BuildKit comparison | Pinned rootless image under gVisor plus ordinary `runc` control | Both controls failed RootlessKit user-namespace setup on this host; no isolation relaxation was adopted | Not a general result for every configured Linux host | +| Collision, input and startup guards | Same-name foreign-volume probe, fake-Docker `../../` evidence-ID probe and `experiments/pilot00_gvisor_offline/test_probe.sh` | Exit 73 left the foreign label/data unchanged and created no harness container; exit 64 made zero Docker calls and created no escaped path; immediate and 45-attempt daemon-readiness paths passed | Concurrent hostile Docker administration is outside this local experiment | +| macOS behavior | Official gVisor platform constraints and honest host inventory | Linux proof only; ordinary Docker Desktop is an explicitly recorded `docker-dev` fallback, not gVisor | No macOS host was available; Apple Silicon and custom VM runtime paths are unverified | + +## Review findings + +- A pre-existing same-name outer volume was reusable before its ownership label was checked. The probe now refuses a foreign volume before mounting or starting the privileged harness and rechecks ownership immediately after creation. +- An operator-supplied evidence ID could escape its results directory. The probe now accepts only a bounded ASCII alphanumeric/hyphen grammar and rejects invalid input before filesystem or Docker access. +- Exhausted inner-daemon readiness checks previously fell through to an opaque runtime-inventory failure. The probe now stops with an explicit bounded-readiness diagnostic, with stubbed immediate-success and 45-failure controls. + +## Reconciliation + +- Current-source reconciliation: Reconciled with `main` + `b169e83f816bba417fc0618a6e2d419acb2f94ed`, retaining the merged local pilot, + checker-delivery and lane-catalogue owners. The spike changes no backend + contract, migration, authority, product composition or workflow. +- Next usable boundary: After review and issue linkage, PILOT-04 can adopt the sealed-cache Kaniko builder and separate gVisor oracle boundary; PILOT-06 still owns checker images and representative fixtures. +- Remaining risks: Actual hosted infrastructure, representative Terminal-Bench resource limits, cleanup after host loss, byte-reproducible image output and macOS Docker Desktop behavior remain unproved. diff --git a/docs/engineering/pilot00-gvisor-offline-build-spike.md b/docs/engineering/pilot00-gvisor-offline-build-spike.md new file mode 100644 index 000000000..b8267c991 --- /dev/null +++ b/docs/engineering/pilot00-gvisor-offline-build-spike.md @@ -0,0 +1,194 @@ +# PILOT-00 offline gVisor build spike + +## Result + +A digest-pinned sample Dockerfile built successfully with the maintained Kaniko +fork inside gVisor with networking disabled, no host container socket and a +non-privileged container configuration. A trusted launcher then loaded the +resulting image tar and ran the oracle in a second gVisor sandbox. The oracle ran +as UID 65532 with no effective capabilities, a read-only root filesystem, only +loopback networking and no Docker socket. + +This establishes feasibility for the small included fixture. It does not +establish production readiness or resource limits for a typical Terminal-Bench +task. The measured builder runs as UID 0 *inside gVisor* with only `CHOWN`, +`DAC_OVERRIDE`, `FOWNER` and `SYS_CHROOT`; its Docker container is not +privileged and does not receive `SYS_ADMIN`. The second oracle sandbox is +non-root and capability-free. + +The development probe uses a privileged, networkless Docker-in-Docker container +as trusted outer infrastructure because the available host cannot register a +new Docker runtime without administrator access. The submitted Dockerfile and +oracle never enter that outer trust boundary directly: they run in inner +`runsc` containers. This harness is a way to test gVisor on a development host, +not evidence that a whole deployment is privilege-free. A hosted launcher needs +a dedicated Linux execution node where an administrator has registered `runsc`; it must +fail closed when that runtime is absent. + +## Recommendation for PILOT-04 and PILOT-06 + +Use the maintained `osscontainertools/kaniko` executor for the first bounded +external image builder, and execute the produced image in a second gVisor +sandbox. Keep image preparation, output import and sandbox launch in the trusted +launcher. Do not expose its Docker or containerd socket to the builder, produced +image, submitted tests or oracle. + +The launch sequence should be: + +1. On trusted infrastructure, resolve each approved base to a platform manifest + digest, fetch it before the attempt and create a digest-bound OCI cache. +2. Mount the request context and sealed cache read-only into a fresh, networkless + Kaniko sandbox. Give it one bounded writable output directory and no secret. +3. Hash the image tar outside the sandbox and bind that attempt-specific digest + to the immutable evaluation request and result contracts selected in + PILOT-04. The spike does not add a transport or persisted field. +4. Import the tar through the trusted launcher and run each oracle/test phase in + a new `runsc` sandbox with a read-only root, non-root UID, no capability, + no socket, no external network and phase-specific CPU, memory, PID, disk and + deadline limits. +5. Preserve the distinction between a checker/work result and launcher, + material or capacity failure. Remove all attempt containers and writable + scratch after evidence has been sealed. + +`canonical-base-image` should select an approved image family and platform. +`pinned-from` should require the exact platform manifest digest that indexes a +present sealed cache entry. An index digest alone is insufficient when the +execution platform has not been fixed. Missing cache material must fail the attempt +as infrastructure; the launcher must not turn networking on to repair it. +Remote `ADD`, package downloads and other network-dependent Dockerfile steps are +therefore unsupported unless all required bytes are supplied as verified input. + +The first launcher limits may use the tested envelope of one CPU, 1 GiB and 128 +PIDs for this builder and 0.5 CPU, 256 MiB and 32 PIDs for this oracle. These are +probe settings, not safe production limits. PILOT-04/PILOT-06 must benchmark an +adjudicated, representative task before fixing defaults. In particular, output +disk needs room for the image tar plus builder scratch and evidence; the 1 MiB +negative probe failed while the sample tar alone was 3,638,784 bytes. + +## Reproducible probe + +The artifact is under +[`experiments/pilot00_gvisor_offline`](../../experiments/pilot00_gvisor_offline/README.md). +On Linux x86_64 with Docker available to the current user: + +```bash +PILOT00_RUN_ID=repro ./experiments/pilot00_gvisor_offline/probe.sh all +PILOT00_RUN_ID=repro ./experiments/pilot00_gvisor_offline/probe.sh cleanup +``` + +`prepare` is the network-using trusted phase. `run` is offline and produces +machine-readable results, launch inspection, logs and point-in-time resource +samples. `cleanup` removes only the labeled outer container and volume for its +run ID. Run IDs accept a lowercase letter or digit plus up to 30 lowercase +letters, digits or hyphens. Optional evidence IDs accept an ASCII letter or +digit plus up to 47 ASCII letters, digits or hyphens. The script validates each +before using it and refuses to mount, start or clean a container or volume +without its ownership label. + +The final Linux replay used these identities: + +| Component | Exact identity | +| --- | --- | +| Host | Ubuntu 24.04, Linux `6.8.0-142-generic`, x86_64, 4 CPUs, 16,377,068 kB memory | +| Docker | client/server `29.1.3`, `overlay2`, systemd cgroups | +| gVisor | `runsc release-20260928.0`, `systrap`; archive SHA-512 `4ce35ca83aef7f96b06cde668e0b23aa98b05aa1829508e974196c2a1e02786c95f5bf79315fd7ddcfd88fe7a00f083ed8053e25eff7673d28d5256440caae8b` | +| Trusted outer harness | `docker@sha256:173f284a4299164772a90f52b373e73e087583c0963f1334c9995f190ef6f3f5` | +| Builder | `ghcr.io/osscontainertools/kaniko@sha256:d6d74217dc077acfd3094992e917c357080a2d3fdd1042a49e34e29a7e57c572`; loaded image ID `sha256:b8c743a43e82f0cef9956ea09eae79c2d56b51d359b65a736c171624a9fc95e3` | +| Base index | `alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc` | +| Base linux/amd64 manifest | `alpine@sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e` | +| Sealed cache manifest | SHA-256 `0d7c48edceae685051d2f60497b0df7f55a0f3f8dd6d29c5e82b9c0cd75a40e1` | + +The final replay evidence ID was `20261008T120000Z`. The result JSON SHA-256 was +`afc06f12ed50606372147a97824467ec448d28037ab8a5baa8dc26616b844c81`. +It recorded: + +| Probe | Result | Elapsed time | +| --- | --- | ---: | +| Offline sample image build | Completed; 3,638,784-byte tar, SHA-256 `0e8a6f4236304340e5bd56c6d45457880c19f27b6bd7994b2b4bbed66797aa29` | 16.420 s | +| Separate oracle sandbox | Passed payload digest, gVisor marker, UID, capability, socket and network checks | 3.226 s | +| Intentional Dockerfile `exit 42` | Genuine work failure; no output image | 17.953 s | +| Empty offline base cache | Infrastructure candidate; registry unavailable with network disabled, exit 1 | 11.030 s | +| Five-second deadline followed by bounded stop | Infrastructure candidate; still running at deadline, exit 143 | 9.645 s including stop | +| 16 MiB builder limit | Infrastructure candidate; cgroup OOM, exit 137 | 3.206 s | +| 1 MiB output filesystem | Infrastructure candidate; `no space left on device`, exit 1 | 17.242 s | + +The one-CPU build samples observed a 102.5% peak from Docker's interval +sampler, 27,283,948 bytes peak memory, 35 PIDs and `0B / 0B` network I/O. The +sealed base cache occupied 3,633,138 bytes and the downloaded/extracted gVisor +tools occupied 457,587,497 bytes. The cache was mounted read-only and its files +were rehashed after the failure probes. The measured numbers cover a tiny +Alpine fixture and cannot be extrapolated to a typical Terminal-Bench task. + +Two identical-input successful builds in the retained replays produced tar SHA-256 values +`22538c93231e0cc76359de6c96ff19b3bebd47ff935d8f8abe317a160e038fc5` +and `0e8a6f4236304340e5bd56c6d45457880c19f27b6bd7994b2b4bbed66797aa29` +because Kaniko records creation timestamps. The trusted launcher can bind and +verify the exact output of each attempt, but this mechanism does not provide +byte-reproducible output across attempts. If PILOT-04 requires reproducible image +bytes rather than request-bound attempt identity, it needs a normalization or +different-builder decision before implementation. + +## Mechanisms rejected by this host probe + +The pinned `moby/buildkit:v0.26.2-rootless` entrypoint did execute in a gVisor +sandbox, but RootlessKit could not install its subordinate UID/GID mappings: + +```text +[rootlesskit:parent] error: failed to setup UID/GID map: +newuidmap ... failed: fork/exec /usr/bin/newuidmap: operation not permitted +``` + +Adding gVisor's `--allow-suid` did not change that result. An ordinary `runc` +control also failed during RootlessKit user-namespace startup on this Ubuntu +host, so the evidence does not prove a BuildKit-specific or gVisor-only defect. +Rootful BuildKit could start its executor only after adding mount/ownership +capabilities that exceed the selected submitted-code boundary. Buildah was not +pursued after the maintained Kaniko path satisfied the bounded sample. + +This result selects Kaniko for the pilot design; it is not a general claim that +rootless BuildKit can never work on another correctly configured Linux node. + +## Failure classification input + +PILOT-04 owns the final contract. The spike supplies these transport-neutral +classification candidates against the existing post-submission result family: + +| Observation | Classification input | +| --- | --- | +| Contributor Dockerfile command exits 42 | Genuine checker/work result, not infrastructure | +| Oracle executes and reports a mismatch | Genuine checker/work result, not infrastructure | +| Digest-bound base absent from sealed cache | `material_unavailable` candidate | +| Attempt crosses its deadline | `deadline_exceeded` candidate | +| Memory, PID or disk quota stops execution | `capacity_exceeded` candidate | +| Registered runtime or builder cannot start | `implementation_unavailable` candidate | +| Launcher receives malformed or oversized checker output | `invalid_output` candidate | + +No-op success, test instability and other checker conclusions also remain +member results. The spike does not define new persisted field names, lifecycle +states, acceptance behavior or retry policy. + +## macOS and deployment limits + +No macOS host was available, so no Docker Desktop runtime evidence was +executed. gVisor runs on Linux; ordinary Docker Desktop containers run in its +Linux VM but are not thereby `runsc` sandboxes. A local macOS fallback may use +the explicitly recorded `docker-dev` isolation mode required by PILOT-04, and +must never report it as gVisor or hosted-grade evidence. A custom Docker Desktop +VM/runtime registration path, Apple Silicon behavior and performance are all +unverified. + +The probe also does not establish hosted execution-node hardening, multi-tenant +isolation, representative task sizing, cache distribution/eviction, retained +image availability, output normalization, concurrency, cleanup after host loss, +or a production launcher. Those remain PILOT-04/PILOT-06 work. + +## Primary references + +- [gVisor installation platforms](https://gvisor.dev/docs/user_guide/install/) +- [gVisor Docker runtime registration](https://gvisor.dev/docs/user_guide/quick_start/docker/) +- [gVisor networking modes](https://gvisor.dev/docs/user_guide/networking/) +- [gVisor rootless operation](https://gvisor.dev/docs/user_guide/rootless/) +- [gVisor production guidance](https://gvisor.dev/docs/user_guide/production/) +- [Docker inside gVisor](https://gvisor.dev/docs/tutorials/docker-in-gvisor/) +- [BuildKit rootless mode](https://github.com/moby/buildkit/blob/master/docs/rootless.md) +- [Maintained Kaniko fork](https://github.com/osscontainertools/kaniko) diff --git a/docs/roadmap_status.md b/docs/roadmap_status.md index 9d67c7825..89ba81bf8 100644 --- a/docs/roadmap_status.md +++ b/docs/roadmap_status.md @@ -228,7 +228,7 @@ cannot be reused as post-submission review-gate evidence. See the | Contribution and compensation truth | **Hidden shared participant plus public policy administration** | ContributionPolicyVersion persistence; lifecycle-audit participant; adapter bindings; public Finance policy administration; REV-04B shared acceptance source storage; CON-03C immutable ContributionRecord/CompensationAward storage; obsolete guide-keyed payment storage and task-local payment fields removed; CON-07 source-neutral submitter participant and complete award sets; REV-04C composes it with FinalAcceptance/TASK effects in the caller transaction | Add authority/evidence and complete-set database closure before production consumption. Only actual Reviews create reviewer records. Add audit/outbox and fulfillment-root/ordinal custody before activation; no public recognition or fulfillment route is live | | Fulfillment, reconciliation, and audit | **Planned** | Shared audit foundations, provider-neutral adapter convention, AUTH-OUTBOX-02 live dispatcher authority, retained phase audit decisions, Celery delivery/recovery scans and CON-02B custody | Feature-specific handlers and authority, conditional award fulfillment, callbacks, idempotent recovery, reconciliation, bounded operational reads, and release controls | | Runtime diagnostics | **Implemented foundation** | Closed structured logging, explicit API/Celery tracing, bounded metrics, safe correlation and optional typed OTLP export | Restrict broker publishers; configure and secure collector/log access, egress, encryption and finite deletion; prove diagnostics during the release drill; no deployed monitoring claim | -| Frontend and pilot | **Local runtime foundation; product pilot planned** | Checkout-isolated Docker Compose API, prefork Celery process, beat, PostgreSQL, Redis and MinIO stack with local Flow-HMAC identities, existing authority bootstrap/grants, configurable loopback ports and project-scoped state; React + Vite + TypeScript stack decision | Complete provider-backed guide compilation and activated-guide scoped task-denial proof on the base stack, implement only stable backed frontend surfaces, run the real internal pilot, repair findings, and complete release drills; Docker Desktop/macOS runtime proof remains | +| Frontend and pilot | **Local runtime foundation; product pilot planned** | Checkout-isolated Docker Compose API, prefork Celery process, beat, PostgreSQL, Redis and MinIO stack with local Flow-HMAC identities, existing authority bootstrap/grants, configurable loopback ports and project-scoped state; a pinned offline sample image build and separate oracle were proved under gVisor; React + Vite + TypeScript stack decision | Implement the external launcher/checker images and benchmark a representative task before choosing production limits; complete provider-backed guide compilation and activated-guide scoped task-denial proof on the base stack, implement only stable backed frontend surfaces, run the real internal pilot, repair findings, and complete release drills; Docker Desktop/macOS runtime proof remains | ## What Has Been Completed @@ -836,6 +836,12 @@ The [local pilot stack record](../.commitrail/changes/pilot-local-stack.md) and authority bootstrap, retained-data boundary and remaining platform/journey proof. +The [offline gVisor build spike](engineering/pilot00-gvisor-offline-build-spike.md) +selects a sealed-cache Kaniko builder plus a separate gVisor oracle sandbox for +PILOT-04/PILOT-06. It proves only the included small Linux fixture; the external +launcher, representative task sizing, hosted hardening and macOS fallback +evidence remain. + Internal chunk identifiers are useful for implementation traceability, but a reader does not need internal engineering records to understand the roadmap above. The main diff --git a/experiments/pilot00_gvisor_offline/README.md b/experiments/pilot00_gvisor_offline/README.md new file mode 100644 index 000000000..e20f67ad3 --- /dev/null +++ b/experiments/pilot00_gvisor_offline/README.md @@ -0,0 +1,55 @@ +# PILOT-00 offline gVisor build probe + +This experiment builds a digest-pinned sample Dockerfile with maintained Kaniko +inside gVisor, then loads the resulting image tar and runs a separate oracle in +a second gVisor sandbox. Both sandboxes have external networking disabled and +receive no Docker or containerd socket. The builder is a non-privileged Docker +container with four capabilities inside gVisor; the oracle runs as UID 65532 +with no capabilities and a read-only root filesystem. + +The Kaniko builder runs as root inside gVisor because it needs filesystem +ownership operations. Its Docker container is not privileged: all capabilities +are dropped and only `CHOWN`, `DAC_OVERRIDE`, `FOWNER` and `SYS_CHROOT` are +added inside the sandbox. The script records those facts before each build. + +The script uses a privileged, networkless Docker-in-Docker container only as a +development harness because an unprivileged user cannot register `runsc` with +the host daemon. That outer harness is trusted infrastructure and is not proof +that the whole deployment is privilege-free. A hosted launcher should instead +run on a dedicated Linux execution node whose administrator has registered `runsc`. + +On Linux x86_64 with Docker available to the current user: + +```bash +PILOT00_RUN_ID=repro ./experiments/pilot00_gvisor_offline/probe.sh all +PILOT00_RUN_ID=repro ./experiments/pilot00_gvisor_offline/probe.sh cleanup +``` + +The bounded shell regression for daemon readiness uses stubs and requires no +Docker resources: + +```bash +./experiments/pilot00_gvisor_offline/test_probe.sh +``` + +`prepare` is the only network-using phase. It downloads the exact gVisor +release, pulls digest-pinned harness/builder/base images, and warms the base +cache. The harness itself starts with `--network none`. `run` mounts the sealed +cache read-only and produces JSON, logs, launch inspection and sampled resource +statistics beneath `/tmp/ws-pilot00-/results/`. `cleanup` removes only +the labeled container and volume for that run ID and retains the evidence. + +`PILOT00_RUN_ID` accepts a lowercase letter or digit followed by at most 30 +lowercase letters, digits or hyphens. The optional `PILOT00_EVIDENCE_ID` +accepts an ASCII letter or digit followed by at most 47 ASCII letters, digits or +hyphens; its default is the UTC timestamp form `YYYYMMDDTHHMMSSZ`. Both values +are rejected before their corresponding resource or evidence path is used. +Harness startup makes 45 bounded inner-daemon readiness attempts and reports a +specific readiness failure before any runtime inventory query. + +The probe also distinguishes an intentional Dockerfile exit from cache-miss, +deadline, memory and output-disk failures. It is deliberately small; its +measurements are a feasibility floor, not Terminal-Bench production sizing. +The BuildKit and base-index entries in `tool-inputs.env` bind the alternatives +examined by the engineering result; the executable success path uses the +platform-specific base manifest and Kaniko pins. diff --git a/experiments/pilot00_gvisor_offline/daemon_ready.sh b/experiments/pilot00_gvisor_offline/daemon_ready.sh new file mode 100644 index 000000000..c03579872 --- /dev/null +++ b/experiments/pilot00_gvisor_offline/daemon_ready.sh @@ -0,0 +1,12 @@ +wait_for_daemon() { + local name="$1" + local attempt + for ((attempt = 1; attempt <= 45; attempt++)); do + if docker exec "${name}" docker info >/dev/null 2>&1; then + return 0 + fi + sleep 1 + done + printf 'inner Docker daemon did not become ready after 45 attempts: %s\n' "${name}" >&2 + return 1 +} diff --git a/experiments/pilot00_gvisor_offline/oracle.sh b/experiments/pilot00_gvisor_offline/oracle.sh new file mode 100755 index 000000000..8c5247d7e --- /dev/null +++ b/experiments/pilot00_gvisor_offline/oracle.sh @@ -0,0 +1,24 @@ +#!/bin/sh +set -eu + +expected_payload='workstream-pilot00-offline-build' +payload_path='/opt/workstream-pilot00/payload.txt' +digest_path='/opt/workstream-pilot00/payload.sha256' + +test "$(id -u)" -ne 0 +test -e /proc/gvisor/kernel_is_gvisor +test ! -e /var/run/docker.sock +test "$(sed -n 's/^CapEff:[[:space:]]*//p' /proc/self/status)" = 0000000000000000 +test "$(cat "$payload_path")" = "$expected_payload" +(cd /opt/workstream-pilot00 && sha256sum -c payload.sha256) + +if wget -T 2 -qO- http://1.1.1.1 >/dev/null 2>&1; then + echo 'external network unexpectedly reachable' >&2 + exit 71 +fi + +interface_count="$(awk 'NR > 2 { count += 1 } END { print count + 0 }' /proc/net/dev)" +test "$interface_count" -eq 1 +grep -Eq '^[[:space:]]*lo:' /proc/net/dev + +printf '%s\n' 'oracle=pass' diff --git a/experiments/pilot00_gvisor_offline/probe.sh b/experiments/pilot00_gvisor_offline/probe.sh new file mode 100755 index 000000000..de970980c --- /dev/null +++ b/experiments/pilot00_gvisor_offline/probe.sh @@ -0,0 +1,572 @@ +#!/usr/bin/env bash +set -euo pipefail + +readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=tool-inputs.env +source "${SCRIPT_DIR}/tool-inputs.env" + +readonly ACTION="${1:-all}" +readonly RUN_ID="${PILOT00_RUN_ID:-repro}" +[[ "${RUN_ID}" =~ ^[a-z0-9][a-z0-9-]{0,30}$ ]] || { + echo "PILOT00_RUN_ID must match [a-z0-9][a-z0-9-]{0,30}" >&2 + exit 64 +} + +readonly PREFIX="ws-pilot00-${RUN_ID}" +readonly DIND_NAME="${PREFIX}-dind" +readonly DIND_VOLUME="${PREFIX}-dind-state" +readonly STATE_DIR="${PILOT00_STATE_DIR:-/tmp/${PREFIX}}" +readonly TOOLS_DIR="${STATE_DIR}/tools" +readonly CACHE_DIR="${STATE_DIR}/cache" +readonly RESULTS_DIR="${STATE_DIR}/results" +readonly DAEMON_CONFIG="${STATE_DIR}/daemon.json" +readonly OWNER_LABEL="org.workstream.pilot00.run=${RUN_ID}" +# shellcheck source=daemon_ready.sh +source "${SCRIPT_DIR}/daemon_ready.sh" + +require_command() { + command -v "$1" >/dev/null 2>&1 || { + echo "missing required command: $1" >&2 + exit 69 + } +} + +assert_linux_amd64() { + [[ "$(uname -s)" == Linux && "$(uname -m)" == x86_64 ]] || { + echo "this recorded probe supports only Linux x86_64" >&2 + exit 69 + } +} + +assert_owned_container() { + local name="$1" + [[ "$(docker inspect --format '{{index .Config.Labels "org.workstream.pilot00.run"}}' "${name}")" == "${RUN_ID}" ]] +} + +assert_owned_volume() { + local name="$1" + [[ "$(docker volume inspect --format '{{index .Labels "org.workstream.pilot00.run"}}' "${name}")" == "${RUN_ID}" ]] +} + +ensure_owned_volume() { + if docker volume inspect "${DIND_VOLUME}" >/dev/null 2>&1; then + assert_owned_volume "${DIND_VOLUME}" || { + echo "refusing volume not owned by this probe: ${DIND_VOLUME}" >&2 + exit 73 + } + else + docker volume create --label "${OWNER_LABEL}" "${DIND_VOLUME}" >/dev/null + assert_owned_volume "${DIND_VOLUME}" || { + echo "refusing volume not owned by this probe: ${DIND_VOLUME}" >&2 + exit 73 + } + fi +} + +prepare_tools() { + mkdir -p "${TOOLS_DIR}" + if [[ ! -f "${TOOLS_DIR}/gvisor.tar.zstd" ]]; then + curl --fail --location --retry 2 --output "${TOOLS_DIR}/gvisor.tar.zstd" "${GVISOR_URL}" + fi + printf '%s %s\n' "${GVISOR_ARCHIVE_SHA512}" "${TOOLS_DIR}/gvisor.tar.zstd" \ + | sha512sum --check + if [[ ! -x "${TOOLS_DIR}/runsc" ]]; then + tar --zstd -xf "${TOOLS_DIR}/gvisor.tar.zstd" -C "${TOOLS_DIR}" + fi + "${TOOLS_DIR}/runsc" --version | grep -F "${GVISOR_VERSION}" >/dev/null +} + +prepare_images_and_cache() { + docker pull "${DIND_IMAGE}" >/dev/null + docker pull "${KANIKO_IMAGE}" >/dev/null + docker pull "${BASE_PLATFORM_IMAGE}" >/dev/null + [[ "$(docker image inspect "${KANIKO_IMAGE}" --format '{{.Id}}')" == "${KANIKO_IMAGE_ID}" ]] + + if [[ ! -s "${CACHE_DIR}/cache.sha256" ]]; then + rm -f "${CACHE_DIR}/cache.sha256" "${CACHE_DIR}/cache.sha256.tmp" + mkdir -p "${CACHE_DIR}/content" + docker run --rm \ + --name "${PREFIX}-warmer" \ + --label "${OWNER_LABEL}" \ + --mount "type=bind,src=${CACHE_DIR}/content,dst=/cache" \ + --entrypoint /kaniko/warmer \ + "${KANIKO_IMAGE}" \ + --cache-dir=/cache \ + --image="${BASE_PLATFORM_IMAGE}" + docker run --rm --network none \ + --name "${PREFIX}-cache-readable" \ + --label "${OWNER_LABEL}" \ + --mount "type=bind,src=${CACHE_DIR}/content,dst=/cache" \ + --entrypoint /busybox/sh \ + "${KANIKO_IMAGE}" \ + -ceu 'chmod -R a+rX /cache' + ( + cd "${CACHE_DIR}/content" + find . -type f -print0 | sort -z | xargs -0 sha256sum > "${CACHE_DIR}/cache.sha256.tmp" + ) + mv "${CACHE_DIR}/cache.sha256.tmp" "${CACHE_DIR}/cache.sha256" + docker run --rm --network none \ + --name "${PREFIX}-cache-seal" \ + --label "${OWNER_LABEL}" \ + --mount "type=bind,src=${CACHE_DIR}/content,dst=/cache" \ + --entrypoint /busybox/sh \ + "${KANIKO_IMAGE}" \ + -ceu 'find /cache -type d -exec chmod 0555 {} \;; find /cache -type f -exec chmod 0444 {} \;' + fi + ( + cd "${CACHE_DIR}/content" + sha256sum --check "${CACHE_DIR}/cache.sha256" + ) + local platform_digest="${BASE_PLATFORM_IMAGE##*@}" + [[ -f "${CACHE_DIR}/content/${platform_digest}/blobs/sha256/${platform_digest#sha256:}" ]] +} + +write_daemon_config() { + cat > "${DAEMON_CONFIG}" <<'JSON' +{ + "runtimes": { + "ws-pilot00-runsc": { + "path": "/opt/gvisor/runsc", + "runtimeArgs": [ + "--network=none", + "--platform=systrap", + "--host-uds=none", + "--gvisor-marker-file" + ] + } + }, + "features": {"containerd-snapshotter": false}, + "iptables": false, + "ip-masq": false, + "bridge": "none" +} +JSON +} + +start_harness() { + mkdir -p "${RESULTS_DIR}" + write_daemon_config + ensure_owned_volume + if docker inspect "${DIND_NAME}" >/dev/null 2>&1; then + assert_owned_container "${DIND_NAME}" || { + echo "refusing container not owned by this probe: ${DIND_NAME}" >&2 + exit 73 + } + if [[ "$(docker inspect --format '{{.State.Running}}' "${DIND_NAME}")" != true ]]; then + docker start "${DIND_NAME}" >/dev/null + fi + else + docker run -d \ + --name "${DIND_NAME}" \ + --label "${OWNER_LABEL}" \ + --privileged \ + --network none \ + --cpus 2 \ + --memory 4g \ + --pids-limit 768 \ + -e DOCKER_TLS_CERTDIR= \ + -v "${DIND_VOLUME}:/var/lib/docker" \ + -v "${TOOLS_DIR}:/opt/gvisor:ro" \ + -v "${DAEMON_CONFIG}:/etc/docker/pilot-daemon.json:ro" \ + -v "${SCRIPT_DIR}:/pilot:ro" \ + -v "${STATE_DIR}:/evidence" \ + "${DIND_IMAGE}" \ + --config-file=/etc/docker/pilot-daemon.json >/dev/null + fi + + wait_for_daemon "${DIND_NAME}" + docker exec "${DIND_NAME}" docker info --format '{{json .Runtimes}}' \ + | grep -F 'ws-pilot00-runsc' >/dev/null + + if ! docker exec "${DIND_NAME}" docker image inspect "${KANIKO_IMAGE_ID}" >/dev/null 2>&1; then + docker save "${KANIKO_IMAGE}" | docker exec -i "${DIND_NAME}" docker load >/dev/null + fi + docker exec "${DIND_NAME}" docker image inspect "${KANIKO_IMAGE_ID}" >/dev/null +} + +inner_docker() { + docker exec "${DIND_NAME}" docker "$@" +} + +create_builder() { + local name="$1" + local context="$2" + local cache_path="$3" + local output_dir="$4" + local output_name="$5" + local destination="$6" + local memory="$7" + local output_mount="${8:-bind}" + local -a output_option + if [[ "${output_mount}" == tmpfs ]]; then + output_option=(--tmpfs /out:rw,size=1m,mode=0700) + else + output_option=(--mount "type=bind,src=${output_dir},dst=/out") + fi + + inner_docker create \ + --name "${name}" \ + --runtime ws-pilot00-runsc \ + --network none \ + --user 0:0 \ + --cap-drop ALL \ + --cap-add CHOWN \ + --cap-add DAC_OVERRIDE \ + --cap-add FOWNER \ + --cap-add SYS_CHROOT \ + --security-opt no-new-privileges \ + --pids-limit 128 \ + --memory "${memory}" \ + --cpus 1 \ + --mount type=bind,src=/pilot,dst=/workspace,readonly \ + --mount "type=bind,src=${cache_path},dst=/cache,readonly" \ + "${output_option[@]}" \ + "${KANIKO_IMAGE_ID}" \ + --dockerfile="/workspace/${context}/Dockerfile" \ + --context="dir:///workspace/${context}" \ + --cache=true \ + --cache-dir=/cache \ + --cache-run-layers=false \ + --cache-copy-layers=false \ + --no-push \ + --no-push-cache \ + --tar-path="/out/${output_name}" \ + --destination="${destination}" \ + --force \ + --log-format=text >/dev/null +} + +capture_container() { + local name="$1" + local result_dir="$2" + local stem="$3" + inner_docker inspect "${name}" > "${result_dir}/${stem}-inspect.json" + inner_docker logs "${name}" > "${result_dir}/${stem}.log" 2>&1 || true +} + +capture_builder_boundary() { + local result_dir="$1" + inner_docker run --rm \ + --name "${PREFIX}-builder-boundary" \ + --runtime ws-pilot00-runsc \ + --network none \ + --user 0:0 \ + --cap-drop ALL \ + --cap-add CHOWN \ + --cap-add DAC_OVERRIDE \ + --cap-add FOWNER \ + --cap-add SYS_CHROOT \ + --security-opt no-new-privileges \ + --pids-limit 128 \ + --memory 1g \ + --cpus 1 \ + --entrypoint /busybox/sh \ + "${KANIKO_IMAGE_ID}" \ + -ceu ' + test "$(id -u)" -eq 0 + test -e /proc/gvisor/kernel_is_gvisor + test ! -e /var/run/docker.sock + capability_mask="$(sed -n "s/^CapEff:[[:space:]]*//p" /proc/self/status)" + test "$capability_mask" = 000000000004000b + if wget -T 2 -qO- http://1.1.1.1 >/dev/null 2>&1; then + exit 71 + fi + interface_count="$(awk "NR > 2 { count += 1 } END { print count + 0 }" /proc/net/dev)" + test "$interface_count" -eq 1 + grep -Eq "^[[:space:]]*lo:" /proc/net/dev + printf "builder_uid=0\nbuilder_cap_eff=%s\nbuilder_network=loopback-only\nbuilder_host_socket=absent\nbuilder_runtime=gvisor\n" "$capability_mask" + ' > "${result_dir}/builder-boundary.log" +} + +run_probe() { + local evidence_id="${PILOT00_EVIDENCE_ID:-$(date -u +%Y%m%dT%H%M%SZ)}" + [[ "${evidence_id}" =~ ^[A-Za-z0-9][A-Za-z0-9-]{0,47}$ ]] || { + echo "PILOT00_EVIDENCE_ID must match [A-Za-z0-9][A-Za-z0-9-]{0,47}" >&2 + exit 64 + } + assert_owned_container "${DIND_NAME}" || { + echo "run prepare first" >&2 + exit 69 + } + local host_result_dir="${RESULTS_DIR}/${evidence_id}" + local inner_result_dir="/evidence/results/${evidence_id}" + local inner_output_dir="${inner_result_dir}/output" + mkdir -p "${host_result_dir}" + docker exec "${DIND_NAME}" mkdir -p "${inner_output_dir}" + docker exec "${DIND_NAME}" chown 0:0 "${inner_output_dir}" + docker exec "${DIND_NAME}" chmod 0755 "${inner_output_dir}" + + capture_builder_boundary "${host_result_dir}" + + local build_name="${PREFIX}-build" + local image_tag="ws-pilot00-sample:${RUN_ID}-${evidence_id,,}" + create_builder "${build_name}" sample /evidence/cache/content "${inner_output_dir}" sample.tar "${image_tag}" 1g + inner_docker inspect "${build_name}" > "${host_result_dir}/build-launch.json" + : > "${host_result_dir}/build-stats.ndjson" + local build_start_ns build_end_ns build_exit + build_start_ns="$(date +%s%N)" + inner_docker start "${build_name}" >/dev/null + while [[ "$(inner_docker inspect "${build_name}" --format '{{.State.Running}}')" == true ]]; do + inner_docker stats --no-stream --format '{{json .}}' "${build_name}" \ + >> "${host_result_dir}/build-stats.ndjson" || true + sleep 0.5 + done + build_end_ns="$(date +%s%N)" + build_exit="$(inner_docker inspect "${build_name}" --format '{{.State.ExitCode}}')" + capture_container "${build_name}" "${host_result_dir}" build + inner_docker rm -v "${build_name}" >/dev/null + [[ "${build_exit}" == 0 ]] + + local output_sha output_bytes + output_sha="$(sha256sum "${host_result_dir}/output/sample.tar" | cut -d' ' -f1)" + output_bytes="$(stat -c %s "${host_result_dir}/output/sample.tar")" + inner_docker load -i "${inner_output_dir}/sample.tar" > "${host_result_dir}/image-load.log" + + local oracle_name="${PREFIX}-oracle" + inner_docker create \ + --name "${oracle_name}" \ + --runtime ws-pilot00-runsc \ + --network none \ + --read-only \ + --user 65532:65532 \ + --cap-drop ALL \ + --security-opt no-new-privileges \ + --pids-limit 32 \ + --memory 256m \ + --cpus 0.5 \ + --mount type=bind,src=/pilot/oracle.sh,dst=/oracle.sh,readonly \ + --entrypoint /oracle.sh \ + "${image_tag}" >/dev/null + local oracle_start_ns oracle_end_ns oracle_exit + oracle_start_ns="$(date +%s%N)" + inner_docker start --attach "${oracle_name}" > "${host_result_dir}/oracle.log" 2>&1 || true + oracle_end_ns="$(date +%s%N)" + oracle_exit="$(inner_docker inspect "${oracle_name}" --format '{{.State.ExitCode}}')" + inner_docker inspect "${oracle_name}" > "${host_result_dir}/oracle-inspect.json" + inner_docker rm -v "${oracle_name}" >/dev/null + [[ "${oracle_exit}" == 0 ]] + grep -Fx 'oracle=pass' "${host_result_dir}/oracle.log" >/dev/null + + local work_name="${PREFIX}-work-failure" + create_builder "${work_name}" sample-invalid /evidence/cache/content "${inner_output_dir}" invalid.tar "${image_tag}-invalid" 1g + local work_start_ns work_end_ns + work_start_ns="$(date +%s%N)" + inner_docker start --attach "${work_name}" > "${host_result_dir}/work-failure.log" 2>&1 || true + work_end_ns="$(date +%s%N)" + local work_exit + work_exit="$(inner_docker inspect "${work_name}" --format '{{.State.ExitCode}}')" + capture_container "${work_name}" "${host_result_dir}" work-failure + inner_docker rm -v "${work_name}" >/dev/null + [[ "${work_exit}" == 42 ]] + grep -F 'intentional contributor Dockerfile failure' "${host_result_dir}/work-failure.log" >/dev/null + + docker exec "${DIND_NAME}" mkdir -p /evidence/empty-cache + docker exec "${DIND_NAME}" chmod 0555 /evidence/empty-cache + local miss_name="${PREFIX}-cache-miss" + create_builder "${miss_name}" sample /evidence/empty-cache "${inner_output_dir}" cache-miss.tar "${image_tag}-cache-miss" 1g + local miss_start_ns miss_end_ns + miss_start_ns="$(date +%s%N)" + inner_docker start --attach "${miss_name}" > "${host_result_dir}/cache-miss.log" 2>&1 || true + miss_end_ns="$(date +%s%N)" + local miss_exit + miss_exit="$(inner_docker inspect "${miss_name}" --format '{{.State.ExitCode}}')" + capture_container "${miss_name}" "${host_result_dir}" cache-miss + inner_docker rm -v "${miss_name}" >/dev/null + [[ "${miss_exit}" != 0 ]] + grep -F 'network is unreachable' "${host_result_dir}/cache-miss.log" >/dev/null + + local timeout_name="${PREFIX}-timeout" + create_builder "${timeout_name}" sample /evidence/cache/content "${inner_output_dir}" timeout.tar "${image_tag}-timeout" 1g + local timeout_start_ns timeout_end_ns timeout_running timeout_exit timeout_oom + timeout_start_ns="$(date +%s%N)" + inner_docker start "${timeout_name}" >/dev/null + sleep 5 + timeout_running="$(inner_docker inspect "${timeout_name}" --format '{{.State.Running}}')" + inner_docker stop --time 1 "${timeout_name}" >/dev/null + timeout_end_ns="$(date +%s%N)" + timeout_exit="$(inner_docker inspect "${timeout_name}" --format '{{.State.ExitCode}}')" + timeout_oom="$(inner_docker inspect "${timeout_name}" --format '{{.State.OOMKilled}}')" + capture_container "${timeout_name}" "${host_result_dir}" timeout + inner_docker rm -v "${timeout_name}" >/dev/null + [[ "${timeout_running}" == true && "${timeout_oom}" == false ]] + [[ "${timeout_exit}" == 143 || "${timeout_exit}" == 137 ]] + + local memory_name="${PREFIX}-memory" + create_builder "${memory_name}" sample /evidence/cache/content "${inner_output_dir}" memory.tar "${image_tag}-memory" 16m + local memory_start_ns memory_end_ns + memory_start_ns="$(date +%s%N)" + inner_docker start --attach "${memory_name}" > "${host_result_dir}/memory-limit.log" 2>&1 || true + memory_end_ns="$(date +%s%N)" + local memory_exit memory_oom + memory_exit="$(inner_docker inspect "${memory_name}" --format '{{.State.ExitCode}}')" + memory_oom="$(inner_docker inspect "${memory_name}" --format '{{.State.OOMKilled}}')" + capture_container "${memory_name}" "${host_result_dir}" memory-limit + inner_docker rm -v "${memory_name}" >/dev/null + [[ "${memory_exit}" == 137 && "${memory_oom}" == true ]] + + local disk_name="${PREFIX}-disk" + create_builder "${disk_name}" sample /evidence/cache/content /out disk.tar "${image_tag}-disk" 1g tmpfs + local disk_start_ns disk_end_ns + disk_start_ns="$(date +%s%N)" + inner_docker start --attach "${disk_name}" > "${host_result_dir}/disk-limit.log" 2>&1 || true + disk_end_ns="$(date +%s%N)" + local disk_exit + disk_exit="$(inner_docker inspect "${disk_name}" --format '{{.State.ExitCode}}')" + capture_container "${disk_name}" "${host_result_dir}" disk-limit + inner_docker rm -v "${disk_name}" >/dev/null + [[ "${disk_exit}" != 0 ]] + grep -F 'no space left on device' "${host_result_dir}/disk-limit.log" >/dev/null + + ( + cd "${CACHE_DIR}/content" + sha256sum --check "${CACHE_DIR}/cache.sha256" > "${host_result_dir}/cache-verification.log" + ) + + BUILD_MS="$(((build_end_ns-build_start_ns)/1000000))" \ + ORACLE_MS="$(((oracle_end_ns-oracle_start_ns)/1000000))" \ + WORK_MS="$(((work_end_ns-work_start_ns)/1000000))" \ + MISS_MS="$(((miss_end_ns-miss_start_ns)/1000000))" \ + TIMEOUT_MS="$(((timeout_end_ns-timeout_start_ns)/1000000))" \ + MEMORY_MS="$(((memory_end_ns-memory_start_ns)/1000000))" \ + DISK_MS="$(((disk_end_ns-disk_start_ns)/1000000))" \ + OUTPUT_SHA="${output_sha}" OUTPUT_BYTES="${output_bytes}" \ + CACHE_BYTES="$(du -sb "${CACHE_DIR}/content" | cut -f1)" \ + TOOLS_BYTES="$(du -sb "${TOOLS_DIR}" | cut -f1)" \ + CACHE_MANIFEST_SHA="$(sha256sum "${CACHE_DIR}/cache.sha256" | cut -d' ' -f1)" \ + DOCKER_VERSION="$(docker version --format '{{.Client.Version}}/{{.Server.Version}}')" \ + DIND_INPUT="${DIND_IMAGE}" KANIKO_INPUT="${KANIKO_IMAGE}" \ + BASE_PLATFORM_INPUT="${BASE_PLATFORM_IMAGE}" GVISOR_INPUT="${GVISOR_VERSION}" \ + STATS_PATH="${host_result_dir}/build-stats.ndjson" \ + WORK_EXIT="${work_exit}" MISS_EXIT="${miss_exit}" \ + TIMEOUT_EXIT="${timeout_exit}" MEMORY_EXIT="${memory_exit}" \ + DISK_EXIT="${disk_exit}" EVIDENCE_ID="${evidence_id}" \ + python3 - "${host_result_dir}/result.json" <<'PY' +import json +import os +import platform +import sys + + +def size_bytes(value): + units = {"GiB": 1024**3, "MiB": 1024**2, "KiB": 1024, "B": 1} + for unit, multiplier in units.items(): + if value.endswith(unit): + return round(float(value.removesuffix(unit)) * multiplier) + raise ValueError(value) + + +stats = [] +with open(os.environ["STATS_PATH"], encoding="utf-8") as stream: + for line in stream: + if line.strip(): + stats.append(json.loads(line)) + +result = { + "schema_version": "pilot00_gvisor_offline_probe_v1", + "evidence_id": os.environ["EVIDENCE_ID"], + "host": {"platform": platform.platform(), "machine": platform.machine()}, + "inputs": { + "docker_client_server": os.environ["DOCKER_VERSION"], + "gvisor": os.environ["GVISOR_INPUT"], + "outer_harness_image": os.environ["DIND_INPUT"], + "builder_image": os.environ["KANIKO_INPUT"], + "base_platform_manifest": os.environ["BASE_PLATFORM_INPUT"], + "cache_manifest_sha256": os.environ["CACHE_MANIFEST_SHA"], + }, + "isolation": { + "trusted_outer_harness_privileged": True, + "trusted_outer_harness_network": "none", + "runtime": "runsc release-20260928.0 systrap", + "network": "none", + "host_socket_mounted": False, + "builder_privileged": False, + "builder_capabilities": ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SYS_CHROOT"], + "oracle_uid": 65532, + "oracle_capabilities": [], + }, + "limits": { + "builder_cpu": 1.0, + "builder_memory_bytes": 1_073_741_824, + "builder_pids": 128, + "oracle_cpu": 0.5, + "oracle_memory_bytes": 268_435_456, + "oracle_pids": 32, + "output_disk_failure_bytes": 1_048_576, + }, + "observed": { + "build_peak_cpu_percent_of_one_core": max(float(item["CPUPerc"].removesuffix("%")) for item in stats), + "build_peak_memory_bytes": max(size_bytes(item["MemUsage"].split(" / ", 1)[0]) for item in stats), + "build_peak_pids": max(int(item["PIDs"]) for item in stats), + "build_network_io_samples": sorted({item["NetIO"] for item in stats}), + "sealed_cache_bytes": int(os.environ["CACHE_BYTES"]), + "gvisor_tools_bytes": int(os.environ["TOOLS_BYTES"]), + }, + "results": { + "build": {"outcome": "completed", "elapsed_ms": int(os.environ["BUILD_MS"])}, + "oracle": {"outcome": "passed", "elapsed_ms": int(os.environ["ORACLE_MS"])}, + "dockerfile_failure": {"class": "work_failure", "exit_code": int(os.environ["WORK_EXIT"]), "elapsed_ms": int(os.environ["WORK_MS"])}, + "cache_miss": {"class": "infrastructure_failure", "exit_code": int(os.environ["MISS_EXIT"]), "elapsed_ms": int(os.environ["MISS_MS"])}, + "timeout": {"class": "infrastructure_failure", "exit_code": int(os.environ["TIMEOUT_EXIT"]), "elapsed_ms": int(os.environ["TIMEOUT_MS"])}, + "memory_limit": {"class": "infrastructure_failure", "exit_code": int(os.environ["MEMORY_EXIT"]), "elapsed_ms": int(os.environ["MEMORY_MS"]), "oom_killed": True}, + "disk_limit": {"class": "infrastructure_failure", "exit_code": int(os.environ["DISK_EXIT"]), "elapsed_ms": int(os.environ["DISK_MS"])}, + }, + "output": { + "tar_sha256": os.environ["OUTPUT_SHA"], + "tar_bytes": int(os.environ["OUTPUT_BYTES"]), + }, +} +with open(sys.argv[1], "w", encoding="utf-8") as stream: + json.dump(result, stream, indent=2, sort_keys=True) + stream.write("\n") +PY + docker exec "${DIND_NAME}" chmod -R a+rX "${inner_result_dir}" + cat "${host_result_dir}/result.json" +} + +cleanup() { + if docker inspect "${DIND_NAME}" >/dev/null 2>&1; then + assert_owned_container "${DIND_NAME}" || { + echo "refusing container not owned by this probe: ${DIND_NAME}" >&2 + exit 73 + } + docker stop --time 20 "${DIND_NAME}" >/dev/null + docker rm "${DIND_NAME}" >/dev/null + fi + if docker volume inspect "${DIND_VOLUME}" >/dev/null 2>&1; then + assert_owned_volume "${DIND_VOLUME}" || { + echo "refusing volume not owned by this probe: ${DIND_VOLUME}" >&2 + exit 73 + } + docker volume rm "${DIND_VOLUME}" >/dev/null + fi + echo "removed ${PREFIX} Docker resources; retained ${STATE_DIR} evidence" +} + +for command in docker curl sha256sum sha512sum tar python3 du stat; do + require_command "${command}" +done +assert_linux_amd64 + +case "${ACTION}" in + prepare) + prepare_tools + prepare_images_and_cache + start_harness + ;; + run) + run_probe + ;; + all) + prepare_tools + prepare_images_and_cache + start_harness + run_probe + ;; + cleanup) + cleanup + ;; + *) + echo "usage: $0 [prepare|run|all|cleanup]" >&2 + exit 64 + ;; +esac diff --git a/experiments/pilot00_gvisor_offline/sample-invalid/Dockerfile b/experiments/pilot00_gvisor_offline/sample-invalid/Dockerfile new file mode 100644 index 000000000..9cb5ce55f --- /dev/null +++ b/experiments/pilot00_gvisor_offline/sample-invalid/Dockerfile @@ -0,0 +1,3 @@ +FROM alpine@sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e + +RUN printf '%s\n' 'intentional contributor Dockerfile failure' >&2 && exit 42 diff --git a/experiments/pilot00_gvisor_offline/sample/Dockerfile b/experiments/pilot00_gvisor_offline/sample/Dockerfile new file mode 100644 index 000000000..2ae7b55ed --- /dev/null +++ b/experiments/pilot00_gvisor_offline/sample/Dockerfile @@ -0,0 +1,8 @@ +FROM alpine@sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e + +COPY payload.txt /opt/workstream-pilot00/payload.txt +RUN sha256sum /opt/workstream-pilot00/payload.txt \ + > /opt/workstream-pilot00/payload.sha256 + +USER 65532:65532 +ENTRYPOINT ["/bin/sh"] diff --git a/experiments/pilot00_gvisor_offline/sample/payload.txt b/experiments/pilot00_gvisor_offline/sample/payload.txt new file mode 100644 index 000000000..b6ee3e8eb --- /dev/null +++ b/experiments/pilot00_gvisor_offline/sample/payload.txt @@ -0,0 +1 @@ +workstream-pilot00-offline-build diff --git a/experiments/pilot00_gvisor_offline/test_probe.sh b/experiments/pilot00_gvisor_offline/test_probe.sh new file mode 100755 index 000000000..77146c9f6 --- /dev/null +++ b/experiments/pilot00_gvisor_offline/test_probe.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +set -euo pipefail + +readonly TEST_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +readonly DAEMON_READY="${TEST_SCRIPT_DIR}/daemon_ready.sh" + +test_immediate_daemon_readiness() ( + readonly DIND_NAME=ws-pilot00-readiness-success-dind + # shellcheck source=daemon_ready.sh + source "${DAEMON_READY}" + local docker_calls=0 + local sleep_calls=0 + docker() { + [[ "$*" == "exec ${DIND_NAME} docker info" ]] || return 99 + docker_calls=$((docker_calls + 1)) + return 0 + } + sleep() { + sleep_calls=$((sleep_calls + 1)) + } + + wait_for_daemon "${DIND_NAME}" + [[ "${docker_calls}" == 1 ]] + [[ "${sleep_calls}" == 0 ]] +) + +test_exhausted_daemon_readiness() ( + readonly DIND_NAME=ws-pilot00-readiness-failure-dind + # shellcheck source=daemon_ready.sh + source "${DAEMON_READY}" + local docker_calls=0 + local sleep_calls=0 + docker() { + [[ "$*" == "exec ${DIND_NAME} docker info" ]] || return 99 + docker_calls=$((docker_calls + 1)) + return 1 + } + sleep() { + sleep_calls=$((sleep_calls + 1)) + } + + local diagnostic diagnostic_file + diagnostic_file="$(mktemp)" + trap 'rm -f "${diagnostic_file}"' EXIT + if wait_for_daemon "${DIND_NAME}" 2>"${diagnostic_file}"; then + echo "expected daemon readiness exhaustion" >&2 + return 1 + fi + diagnostic="$(cat "${diagnostic_file}")" + [[ "${diagnostic}" == \ + "inner Docker daemon did not become ready after 45 attempts: ws-pilot00-readiness-failure-dind" ]] + [[ "${docker_calls}" == 45 ]] + [[ "${sleep_calls}" == 45 ]] +) + +test_immediate_daemon_readiness +test_exhausted_daemon_readiness +echo "probe readiness tests passed" diff --git a/experiments/pilot00_gvisor_offline/tool-inputs.env b/experiments/pilot00_gvisor_offline/tool-inputs.env new file mode 100644 index 000000000..b4632f5f9 --- /dev/null +++ b/experiments/pilot00_gvisor_offline/tool-inputs.env @@ -0,0 +1,9 @@ +GVISOR_VERSION=release-20260928.0 +GVISOR_ARCHIVE_SHA512=4ce35ca83aef7f96b06cde668e0b23aa98b05aa1829508e974196c2a1e02786c95f5bf79315fd7ddcfd88fe7a00f083ed8053e25eff7673d28d5256440caae8b +GVISOR_URL=https://storage.googleapis.com/gvisor/releases/release/20260928.0/x86_64/gvisor.tar.zstd +BUILDKIT_IMAGE=moby/buildkit@sha256:0ffa2fcf6b8757c47d569b3ef0f03f9d5eb3b9ff5ce68d858f994f89b749da0c +DIND_IMAGE=docker@sha256:173f284a4299164772a90f52b373e73e087583c0963f1334c9995f190ef6f3f5 +KANIKO_IMAGE=ghcr.io/osscontainertools/kaniko@sha256:d6d74217dc077acfd3094992e917c357080a2d3fdd1042a49e34e29a7e57c572 +KANIKO_IMAGE_ID=sha256:b8c743a43e82f0cef9956ea09eae79c2d56b51d359b65a736c171624a9fc95e3 +BASE_IMAGE=alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc +BASE_PLATFORM_IMAGE=alpine@sha256:c64c687cbea9300178b30c95835354e34c4e4febc4badfe27102879de0483b5e