Skip to content

Add a host image for POSIX builds and the QA gates #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hacker-cb merged 1 commit into from
Aug 23, 2026
Merged

Add a host image for POSIX builds and the QA gates #23

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/dependabot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@
# ecosystem resolves.
# - IDF_BASE_TAG, ESP_MATTER_VERSION, PIO_VERSION. These are chosen against
# Espressif's own compatibility matrix, not against "newest".
# - Everything images/host/Dockerfile pins. Its pip installs are ARG defaults,
# which no ecosystem resolves; PAHO_REF is a commit fetched by a `git fetch`
# line; and LYCHEE_VERSION travels with two literal checksums that a bump has
# to recompute per architecture. All three are bumped by hand, and the QA
# versions deliberately track what the consuming project pins for the same
# tools rather than what is newest - the image agreeing with its consumer is
# the whole point of it.
#
# Note that a bump here edits a workflow file, which is in that workflow's own
# `paths:` filter - so every Dependabot PR triggers a full paid rebuild, and
Expand Down
234 changes: 234 additions & 0 deletions .github/workflows/host.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,234 @@
# Host Docker Build Workflow
#
# Build Strategy:
# - Main repo master: Build + Push to GHCR
# - Main repo dev/PR: Build only (validation) - nothing is uploaded. Like
# platformio.yml and unlike esp-idf.yml, no other image is built FROM this one,
# so there is nothing for a pull request to push for.
# - workflow_dispatch: Build on any branch of the main repo, push only on master
# - Forks: nothing runs - the runner pools are not reachable there; use
# ./scripts/build.sh instead
# - Multi-platform: Platform matrix builds linux/amd64 and linux/arm64 in parallel,
# each on a runner of its own architecture (matrix.runner) - no QEMU. This image
# is the one a developer on Apple Silicon runs interactively, so an emulated
# arm64 leg would be the slow half of exactly the audience it exists for.
name: 🐳 Host Docker Image

on:
push:
branches: [master, dev]
# Negations come last and are order-sensitive; `paths:` and `paths-ignore:`
# cannot both be used for one event. Without the exclusion a
# documentation-only commit rebuilds and republishes the image.
paths:
- '.github/workflows/host.yml'
- 'images/host/**'
- 'images/versions.json'
- 'scripts/versions-matrix.sh'
- 'scripts/check-versions.sh'
- '!images/**/*.md'
pull_request:
branches: [master, dev]
paths:
- '.github/workflows/host.yml'
- 'images/host/**'
- 'images/versions.json'
- 'scripts/versions-matrix.sh'
- 'scripts/check-versions.sh'
- '!images/**/*.md'
workflow_dispatch:

env:
REGISTRY: ghcr.io
HOST_IMAGE_NAME: jethome-dev-host
# build-push-action v6+ uploads a build record as a workflow artifact by default.
# This repository is public, so that archive - full build log and metadata - would
# be downloadable by anyone off the run page, on every leg of every push. The
# summary itself is kept: it reports what was built without opening the log, and
# costs no storage.
DOCKER_BUILD_RECORD_UPLOAD: false

# Superseded pull-request runs are cancelled - nothing has been published yet and
# they cost money. Pushes are deliberately NOT grouped together: the group key
# includes the commit, so each one gets its own group and none can cancel another.
#
# `cancel-in-progress: false` would not have given a queue. GitHub cancels a
# *pending* run in a group whenever a newer one arrives, regardless of that flag -
# so grouping master pushes by ref would silently drop the middle commit of any
# three that land inside one build window, and its sha-<short-commit> image, which the
# READMEs document as the way to pin an exact commit, would never exist.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name == 'pull_request' && 'pr' || github.sha }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
# Both matrices come from images/versions.json, so a version lives in one place
# instead of two matrix blocks that have to be edited together. The consistency
# check runs first and fails the whole workflow.
prepare:
name: 🧮 Resolve matrices
runs-on: ubuntu-latest
timeout-minutes: 5
if: github.repository_owner == 'jethome-iot' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
permissions:
contents: read
outputs:
build: ${{ steps.matrices.outputs.build }}
manifest: ${{ steps.matrices.outputs.manifest }}
steps:
- name: 📥 Checkout repository
uses: actions/checkout@v7

- name: ✅ Check version consistency
run: ./scripts/check-versions.sh

- name: 🧮 Generate matrices
id: matrices
run: |
set -euo pipefail
# Assigned first, echoed second: `echo "k=$(cmd)"` takes echo's exit
# status, so a failing generator would write an empty matrix and leave
# this step green.
build=$(./scripts/versions-matrix.sh host build)
manifest=$(./scripts/versions-matrix.sh host manifest)
{
echo "build=${build}"
echo "manifest=${manifest}"
} >> "$GITHUB_OUTPUT"

host-build:
name: host-build (${{ matrix.tag }}, ${{ matrix.platform }})
needs: prepare
runs-on: ${{ matrix.runner }}
timeout-minutes: ${{ matrix.timeout_minutes }}
# Owner-only, with no workflow_dispatch escape hatch: the runner pools belong
# to jethome-iot, a fork cannot resolve their labels, and an unresolvable
# `runs-on` queues for 24 hours rather than failing. A fork that wants to build
# this image runs ./scripts/build.sh.
# Two conditions, not one. The owner check keeps this out of forks that run the
# workflow themselves. The head-repo check keeps it out of pull requests *from*
# a fork, where github.repository_owner is still jethome-iot - so without it a
# fork-controlled Dockerfile would execute on this org's paid runner pools.
if: github.repository_owner == 'jethome-iot' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository)
permissions:
contents: read
packages: write
strategy:
fail-fast: false
matrix:
include: ${{ fromJSON(needs.prepare.outputs.build) }}
steps:
- name: 📥 Checkout repository
uses: actions/checkout@v7

- name: 🔧 Set up Docker Buildx
uses: docker/setup-buildx-action@v4

- name: 🔐 Log in to GitHub Container Registry
if: github.ref_name == 'master'
uses: docker/login-action@v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

# Pushed by digest, with no tag of its own: a tag here would be a mutable
# name that the manifest job below has to look up again later, and between
# those two moments another run can overwrite it. The digest is the only
# thing handed forward.
- name: 🐳 Build and push by digest
id: build
uses: docker/build-push-action@v7
with:
context: images/host
platforms: ${{ matrix.platform }}
outputs: type=image,name=${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.HOST_IMAGE_NAME }},push-by-digest=true,name-canonical=true,push=${{ github.ref_name == 'master' }}
build-args: ${{ matrix.build_args }}

# Matrix legs cannot each set a job output - they would overwrite one
# another - so the digests travel as one empty file per leg, named after the
# digest itself.
- name: 📤 Export digest
if: github.ref_name == 'master'
run: |
mkdir -p /tmp/digests
digest="${{ steps.build.outputs.digest }}"
touch "/tmp/digests/${digest#sha256:}"

- name: ⬆️ Upload digest
if: github.ref_name == 'master'
uses: actions/upload-artifact@v7
with:
name: digest-${{ env.HOST_IMAGE_NAME }}-${{ matrix.tag }}-${{ matrix.platform_tag }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1

host-manifest:
name: host-manifest (${{ matrix.tag }})
runs-on: ubuntu-latest
timeout-minutes: 10
needs: [prepare, host-build]
# See the note in esp-idf.yml: one variant's failure must not withhold another
# variant's tags.
if: >-
${{ !cancelled()
&& needs.prepare.result == 'success'
&& github.repository_owner == 'jethome-iot'
&& github.ref_name == 'master' }}
permissions:
contents: read
packages: write
strategy:
fail-fast: false
matrix:
include: ${{ fromJSON(needs.prepare.outputs.manifest) }}
steps:
- name: 🔐 Log in to GitHub Container Registry
uses: docker/login-action@v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: ⬇️ Download digests
uses: actions/download-artifact@v8
with:
pattern: digest-${{ env.HOST_IMAGE_NAME }}-${{ matrix.tag }}-*
merge-multiple: true
path: /tmp/digests

- name: 🐳 Create and push multi-arch manifest
env:
IMAGE: ${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.HOST_IMAGE_NAME }}
TAG: ${{ matrix.tag }}
PRIMARY: ${{ matrix.primary }}
EXPECTED_PLATFORMS: ${{ matrix.platform_count }}
run: |
set -euo pipefail
cd /tmp/digests
found=$(find . -type f | wc -l | tr -d ' ')
if [ "${found}" -ne "${EXPECTED_PLATFORMS}" ]; then
echo "::error::expected ${EXPECTED_PLATFORMS} platform digests, found ${found}"
exit 1
fi
# Every source is a digest this run produced, so nothing here can be
# resolved to another run's image. Built as an array rather than an
# unquoted expansion so the refs survive as separate arguments without
# relying on word splitting.
refs=()
for digest in *; do
refs+=("${IMAGE}@sha256:${digest}")
done
# Only the primary variant moves `latest` and the bare `sha-<short-commit>`.
# Every variant gets `<version>-sha-<short-commit>`: the version tag itself is
# rewritten on every push, so without it a consumer pinned to a
# non-primary variant has no name for the build they were running.
SHA_SHORT="${GITHUB_SHA:0:7}"
tags=()
if [ "${PRIMARY}" = "true" ]; then
tags+=(-t "${IMAGE}:latest" -t "${IMAGE}:sha-${SHA_SHORT}")
fi
tags+=(-t "${IMAGE}:${TAG}-sha-${SHA_SHORT}")
tags+=(-t "${IMAGE}:${TAG}")
docker buildx imagetools create "${tags[@]}" "${refs[@]}"
46 changes: 31 additions & 15 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,15 @@ This repo ships no application code. It is a home for Docker developer images:
each image lives in `images/<name>/` and is published as
`ghcr.io/<owner>/jethome-dev-<image>`. The deliverables are Dockerfiles, GitHub
Actions workflows and READMEs. `images/` is the roster — today `esp-idf`,
`esp-matter` and `platformio`, and adding to it is the normal case, not an
`esp-matter`, `host` and `platformio`, and adding to it is the normal case, not an
exception. Rules below are written per image; where they name one, it is an
example.

## Layout

- One directory per image: `images/<name>/`, holding its `Dockerfile`, `README.md`
and any support files (`esp-matter/entrypoint.sh`, `platformio/pio_project/`).
and any support files (`esp-matter/entrypoint.sh`, `platformio/pio_project/`,
`host/smoke/`).
- Each workflow sets `context: images/<name>` and `scripts/build.sh` builds the
same directory, so a Dockerfile can only `COPY` from inside its own directory —
a shared file at the repo root breaks both CI and local builds.
Expand All @@ -20,11 +21,13 @@ example.

## CI

The authoritative files are `.github/workflows/esp-idf.yml` and
`.github/workflows/platformio.yml` — read them before changing anything here.
The authoritative files are `.github/workflows/esp-idf.yml`,
`.github/workflows/platformio.yml` and `.github/workflows/host.yml` — read them
before changing anything here.

- One workflow file per image *family*; today there are two — `esp-idf.yml` builds
**both** esp-idf and esp-matter, `platformio.yml` builds platformio. A family is:
- One workflow file per image *family*; today there are three — `esp-idf.yml`
builds **both** esp-idf and esp-matter, `platformio.yml` builds platformio,
`host.yml` builds host. A family is:
an image whose Dockerfile is `FROM` another image of this repo joins the base
image's workflow, adds `images/<name>/**` to its push **and** pull_request
`paths:` filters, and chains via `needs: <base>-build` — the *build* job, not the
Expand Down Expand Up @@ -97,7 +100,7 @@ The authoritative files are `.github/workflows/esp-idf.yml` and
runtime `runs-on` is invisible to actionlint); every `base_tag` exists among the
base image's tags; and the base image's `ARG` default names the primary variant's
base tag. Run it locally with `./scripts/check-versions.sh`.
- The checker validates the whole file and both workflows gate on it, so a version
- The checker validates the whole file and every build workflow gates on it, so a version
file broken for one image blocks publishing for all of them. That is deliberate —
the data is shared — but it is also why `images/versions.json` sits in every
`paths:` filter: a bump to one image rebuilds the others. The alternative, a file
Expand All @@ -118,7 +121,10 @@ The authoritative files are `.github/workflows/esp-idf.yml` and
- An `ARG` with no matrix entry is a pin CI never passes, so its Dockerfile
default is the sole source of truth and is bumped there — today
`ESP32_PLATFORM_VERSION`, `NATIVE_PLATFORM_VERSION` and `UNITY_VERSION` in
`images/platformio/Dockerfile`.
`images/platformio/Dockerfile`, and every tool pin in `images/host/Dockerfile`
(the QA versions, `PAHO_VERSION`/`PAHO_REF`, `LYCHEE_VERSION` and its two
checksums). host passes one arg and one only, `UBUNTU_BASE_TAG`, because that is
the single value its tag can name.
- Every matrix carries `fail-fast: false`, so one platform leg failing does not
cancel the other and truncate its log.
- `concurrency` cancels superseded **pull-request** runs and groups nothing else:
Expand All @@ -144,9 +150,9 @@ The authoritative files are `.github/workflows/esp-idf.yml` and
- **Whether the image is uploaded** differs per image. `esp-idf-build` pushes by
digest on *every* run — its GHCR login is unconditional to match — because
`esp-matter-build` consumes that digest and that is what makes ESP-Matter
validatable on a pull request. `platformio-build` and `esp-matter-build` keep
`push=${{ github.ref_name == 'master' }}`: nothing consumes them, so a PR
builds and throws away.
validatable on a pull request. `platformio-build`, `host-build` and
`esp-matter-build` keep `push=${{ github.ref_name == 'master' }}`: nothing
consumes them, so a PR builds and throws away.

The org literal is hardcoded throughout.
- **Every image is build-validated on pull requests**, ESP-Matter included, and on
Expand All @@ -165,7 +171,7 @@ The authoritative files are `.github/workflows/esp-idf.yml` and
the owner check alone would let a fork-controlled Dockerfile run on this org's
paid pools. The condition is
`github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository`.
- Both workflows exclude `!images/**/*.md` from their `paths:`. Negations come last
- Every build workflow excludes `!images/**/*.md` from its `paths:`. Negations come last
and are order-sensitive, and `paths:` cannot be mixed with `paths-ignore:` for one
event. Without this a documentation-only commit rebuilds and republishes every
image.
Expand Down Expand Up @@ -245,7 +251,14 @@ The authoritative files are `.github/workflows/esp-idf.yml` and
`/opt/esp/python-packages.txt`, a file rather than a pipe so nothing swallows a
failure, and **an image that installs on top regenerates it**: esp-matter's
`install.sh` populates the same venv, so inheriting the base's snapshot would
leave the file describing an environment that no longer exists.
leave the file describing an environment that no longer exists. host takes the
same rule one step further, because a version print cannot reach what it
promises: its layer asserts that `clang-format` and `clang-tidy` *report* the
numbers pip was told to install (the wrapper and the wheel are two different
things), and then configures, builds and `ctest`s `images/host/smoke/` — two
files that prove `find_package(GTest)` resolves, that GMock links, and that the
paho which loads reports the version pinned beside its commit. Its own freeze
goes to `/opt/qa-packages.txt`.
- **A version an image installs by name is a version this repository chose**, and
`./scripts/check-pins.sh` enforces that per package manager, because the price of
a pin differs per manager. `pip` takes `==` on every name — pip here runs with no
Expand Down Expand Up @@ -281,13 +294,16 @@ The authoritative files are `.github/workflows/esp-idf.yml` and
- Docker's default `SHELL` is `/bin/sh`, so an image that sets none must keep its
`RUN` layers POSIX — check the image's own Dockerfile before reaching for a
bash-ism. esp-idf and esp-matter set `SHELL ["/bin/bash", "-c"]` because their
layers call the bash builtin `source`; platformio does not. A trailing
layers call the bash builtin `source`; platformio and host do not — host's
multi-line layers are POSIX `sh` on purpose (`set -eu`, `case`, no arrays). A trailing
`CMD ["/bin/bash"]` sets the interactive shell, not the build shell.
- Toolchain activation is per-image, not repo-wide. The ESP images source
`${IDF_PATH}/export.sh` in every `RUN` that needs the toolchain (esp-idf
additionally needs `export IDF_PATH_FORCE=1` in the same `RUN`; esp-matter also
sources `${ESP_MATTER_PATH}/export.sh`), while platformio puts `pio` on `PATH`
at install time and activates nothing. Either way call tools by name, never by
at install time and activates nothing, and host puts its venv (`/opt/qa-venv`)
first on `PATH` so `ruff`/`mypy`/`clang-tidy` resolve without activation —
Ubuntu 24.04 ships PEP 668, so there is no system interpreter to install into. Either way call tools by name, never by
absolute path: `/opt/esp/python_env/idf5.4_py3.12_env/bin/python3` was tried and
reverted — it pins a version-stamped directory that a version bump invalidates.
- esp-matter activates both environments at runtime through
Expand Down
Loading