Repository navigation
feat: more bread - #1
Merged
Merged
Conversation
restructure repo to act as both a worked example of spread + a host for prebuilt docker images. two image flavours, each across ubuntu 24.04 / 25.10 / 26.04 x amd64 / arm64: - bread: lean ubuntu + sshd only. - chisel-releases-bread: built on top of bread, adds chisel v1.4.1 + the shell + container tooling chisel-releases spread tests expect. top-level produces images + per-flavour, per-version distribution yamls (templates/ -> hack/inline_scripts.py -> inlined/). makefile uses pattern rules + hash-stamp pattern (hack/hash_inputs.sh) so rebuilds only fire when relevant Dockerfile / scripts content actually changed. existing bread example moved into demo/ -- self-contained inlined spread.yaml + tests, still consuming the lean bread images. ghcr publication deferred.
spread's system-name validator (validSystem regex) requires names to start with a letter. bare "24.04-amd64" gets rejected as invalid. add the "ubuntu-" prefix everywhere systems are listed (templates, demo, regenerated inlined yamls) and bump the cut indices in the allocate scripts accordingly (ver = field 2, arch = field 3 once "ubuntu-" is the first field).
three related changes to the makefile: - new top-level demo target. ensures the lean bread images the demo needs are built, then delegates to demo/makefile's run target. demo is pinned to LTS only (24.04 + 26.04 x both arches) so non-LTS images aren't pulled into the demo dependency chain. - env-var narrowing on build-bread / build-chisel-releases-bread: setting VER and/or ARCH narrows the matrix; unset means the full matrix. obsoletes the prior build-bread-% pattern aliases. - extracted the inline shell from the stamp rules into hack/build_image.sh. one helper handles both flavours via case dispatch. makefile stamp rules now one-liners.
move docker.io from the chisel-releases-bread layer into the base bread image. motivation: a bread container can now host nested docker workloads (relevant for upcoming test flow where a bread container acts as the outer host for a spread-in-spread run). also drops the "lean" framing from comments and README -- bread is just the base image, not advertised as minimal. bread image size grows by ~340MB (docker.io + deps). chisel-releases-bread net size unchanged (docker.io install removed, inherited from base instead).
flavour names now share a common bread- prefix, which lines up with the new bread-test test-host image landing in a follow-up commit and groups all bread-family images together when listed alphabetically. pure renaming change: files renamed via git mv, identifier rewritten in every referencing file. no functional change.
undoes commit 97c8e55 (feat: ship docker.io in bread base). the test-host image landing in a follow-up commit installs docker.io itself, so the base bread image doesn't need it after all -- keeps the base small and the docker.io concern scoped to images that genuinely need it. bread back to sshd-only. bread-chisel-releases re-adds docker.io to its apt-install line. dockerfile comments realigned with the new bread-chisel-releases naming.
new image flavour bread-test, only built at 26.04 x both arches. used
as the outer system in upcoming spread-in-spread test flow.
multi-stage Dockerfile (tests/Dockerfile.bread-test-26.04):
- stage 1 (golang:1-bookworm): clones canonical/spread + builds the
spread binary statically (CGO disabled).
- stage 2 (FROM bread:${BASE_TAG}): apt-installs docker.io for
nested docker workloads, copies the spread binary in.
wired through hack/hash_inputs.sh + hack/build_image.sh (case for the
new flavour) and makefile (BREAD_TEST_STAMPS, build-bread-test target,
stamp pattern rule, clean entry). final image lives outside the main
matrix so build-all / clean / etc treat it as an extra.
two related sshd hardening tweaks to the base bread Dockerfiles: 1. append PasswordAuthentication yes to /etc/ssh/sshd_config. ubuntu's default config has the line commented out, and on some recent OpenSSH builds the compile-time default is no rather than yes, so spread's password auth needs the override to land reliably. 2. replace /etc/pam.d/sshd with a pam_permit-only stack. the default pam_unix stack calls unix_chkpwd, which on some hosts (kernel + docker + libxcrypt combo) fails inside privileged containers with "libcrypt.so.1: cannot change memory protections" and returns PAM_PERM_DENIED. these are throwaway test containers; we don't care about password validity, only that spread can attach. inherited by bread-chisel-releases (FROM bread:VER-ARCH) and the bread-test image, so the fix applies everywhere sshd lives.
new tests/ tree exercising the published inlined yamls end-to-end.
outer layer: tests/spread.yaml uses an adhoc backend that spawns a
bread-test container as its sole system (one per arch). the container
gets the docker socket bind-mounted in plus the repo bind-mounted at
/spread-bread, so the outer task can reach both the host docker
daemon and the inlined yamls / contract task definitions.
inner layer: each outer task copies one inlined yaml into a temp
project dir, drops in the matching contract tasks, and runs spread.
the inner spread allocates bread or bread-chisel-releases containers
via the same scripts shipped to consumers, so the test exercises the
distribution artefact literally rather than via a mock.
contract assertions:
- bread: /etc/os-release VERSION_ID matches expected ubuntu version,
uname -m matches expected arch.
- bread-chisel-releases: the above + chisel --version matches the
pinned tag + curl/wget/git/jq/file/sudo/tree/docker/skopeo on PATH.
variants drive each suite over the three ubuntu versions.
makefile wiring + bread-test image build pipeline land separately.
bread-chisel-releases shipped ubuntu's apt docker.io (built with
go 1.24) and bread-test built spread inside a multi-stage docker
build. on hosts where amd64 containers run under qemu, the apt
docker.io's go 1.24 runtime crashes inside the emulated container.
moving every go binary we bake into images onto a single, recent-go
build path fixes that.
new flow:
- hack/build_binaries.sh runs Canonical's ubuntu/go:1.25-26.04_edge
container once. it cross-compiles chisel + spread for both arches
with go 1.25.8+ (GOTOOLCHAIN=auto fetches the toolchain go.mod
demands) and fetches the upstream docker static tarball for each
arch (built with go 1.26.3 -- qemu-safe). chisel version is
injected via -X ldflag so the binary self-reports the pinned tag
rather than "unknown".
- outputs land in cache/binaries/{chisel,spread,docker}-{amd64,arm64}
(gitignored).
- .stamp/binaries gates the build via the usual hash-of-inputs
pattern (CHISEL_REF, SPREAD_REF, DOCKER_VERSION, GO_BUILDER_IMAGE,
plus build_binaries.sh content).
- bread-chisel-releases and bread-test stamps now depend on
.stamp/binaries, so any binary refresh cascades into image
rebuilds.
Dockerfile changes:
- bread-chisel-releases-* drop docker.io from the apt line, COPY
chisel + docker from cache/binaries via a BUILD_ARCH build-arg.
- bread-test-26.04 drops its golang multi-stage builder, COPY's
spread + docker from cache/binaries.
hack/build_image.sh passes --build-arg BUILD_ARCH=$arch through for
both chisel-releases and test images.
each outer arch now runs 2 spread-bread test containers in parallel instead of serialising the 6 jobs through one. inner spread inside each outer is unchanged (still workers: 2 inside its own inlined yaml). end-to-end full-matrix test wall clock drops from ~60s to ~20s on a qcom arm64 dev box.
BREAD_TEST_STAMPS used the full ARCHES list rather than SELECTED_ARCHES, so make build-bread-test ARCH=amd64 still built both arches. consistent narrowing with the other stamp lists; allows the ci build-and-test workflow to run per-arch jobs that only build their own arch bread-test image.
five workflows modelled on the not-quite-rust-rock layout but adapted
for docker images and our two-flavour matrix.
reusable building blocks:
- binaries.yaml: builds cache/binaries via the ubuntu/go cross-compile
container once, uploads the 6 binaries + matching .stamp/binaries
as a single artefact. downstream jobs restore them and avoid the
expensive cross-compile.
- build-and-test.yaml: matrix per-arch on native runners
(ubuntu-24.04 + ubuntu-24.04-arm). each job restores binaries,
builds bread / bread-chisel-releases / bread-test images for its
own arch, installs spread, and runs the spread-in-spread tests
scoped to outer:ubuntu-26.04-<arch>. no qemu in ci.
- publish-ghcr.yaml: per-arch jobs save 6 image tarballs each via
docker save. a single stitch job downloads both arches and uses
buildah manifest to assemble + push multiarch manifests for the
six (flavour, ver) pairs to ghcr.io/<repo>/<flavour>:<ver>. no
per-arch tags published. bread-test is intentionally not
published (test-infra-only).
entry workflows:
- ci.yaml: PR + push to main + manual dispatch. binaries followed
by build-and-test. concurrency cancel-in-progress on the same
ref.
- release.yaml: triggers on push of an r[0-9]+ tag (or manual
dispatch). binaries + build-and-test + publish-ghcr + a
publish-release job that regenerates inlined yamls, deletes any
prior r* releases (rolling: only the latest stays on the GitHub
side), and attaches inlined/*.yaml to the new release.
each inlined yaml lists both ubuntu arches, so before this patch a ci per-arch job would spawn the missing-arch inner container too -- host docker daemon has only the arch we built locally, and the pull fallback fails with denied access. derive arch from SPREAD_SYSTEM and pass a spread filter expression (docker:ubuntu-...-arch:...) to inner spread so it only schedules jobs whose system matches the outer's arch. locally with make all (both arches built), behaviour is unchanged -- the filter just narrows from 2 inner systems to 1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
restructures spread-bread into a docker image host + a worked example + a test harness, and wires up ci/release to publish to ghcr.
image flavours
two flavours, three ubuntu versions x amd64/arm64, published as multiarch on tag push:
layout
top-level produces images + per-flavour, per-version
inlined/*.yamldistribution artefacts (generated fromtemplates/*.yaml.inbyhack/inline_scripts.py, which splices the allocate/discard scripts inline).demo/is a worked bread example.tests/runs the spread-in-spread (spreadception) tests.baked go binaries
chisel + spread cross-compiled w/ go 1.25.8+ in the Canonical ubuntu/go:1.25-26.04_edge container; docker CLI fetched from docker.com static (go 1.26.3). all baked binaries are recent-go so they survive qemu emulation -- ubuntu's apt-shipped docker.io is go 1.24 and crashes inside an amd64-on-arm64 container. cache lives in
cache/binaries/(gitignored).pins live in
makefile(CHISEL_REF, SPREAD_REF, DOCKER_VERSION, GO_BUILDER_IMAGE).tests
tests/spread.yamlis an outer spread that allocates abread-testcontainer (bread + docker + spread, test-infra only, not published). each outer task copies one inlined yaml into a temp project + drops in a matching contract task, then runs an innerspread. the inner allocates real bread / bread-chisel-releases containers via the same scripts shipped to consumers, so the test exercises the published artefact literally rather than via a mock.inner contract: ubuntu VERSION_ID matches expected, uname -m matches expected, and for the chisel flavour
chisel --versionmatches the pinned tag + presence of the expected tools.makefile
hash-stamp pattern; rebuilds fire only on real input changes. env-var narrowing for partial builds:
make build-bread VER=24.04 ARCH=amd64make build-bread-chisel-releases ARCH=arm64make demoruns the demo;cd tests && spreadruns the contract suite.ci / release
ci.yaml: PR + push to main. binaries -> build-and-test on per-arch native runners (ubuntu-24.04 + ubuntu-24.04-arm). no qemu, no publish.release.yaml: push ofr[0-9]+tag. binaries -> build-and-test -> publish-ghcr + publish-release. multiarch tags assembled viabuildah manifestfromdocker savetarballs (no per-arch tags published). github release rolls -- prior r* releases get deleted, only the latest stays;inlined/*.yamlattached as assets.