From 946fffb3047d1ba33c4c549f45250c7634d34ef7 Mon Sep 17 00:00:00 2001 From: Pavel Sokolov Date: Sun, 23 Aug 2026 10:25:51 +0300 Subject: [PATCH] Add a host image for POSIX builds and the QA gates A static analyzer reads the host's standard library, not the project's: libc++ inlines the throw inside std::string and std::function, libstdc++ hides them behind external __throw_*, and the same pinned clang-tidy therefore reports findings on macOS that CI never sees - and misses code that is only red on a laptop. This image is the Linux answer, buildable and runnable on either architecture. It also replaces the apt-plus-build-paho preamble a consuming CI job runs before every POSIX leg: build-essential, CMake, Ninja, ccache and GTest/GMock, paho.mqtt.c built from a pinned commit with the same flags that job used, and the QA toolchain (ruff, mypy, pytest, jsonschema, clang-format, clang-tidy, lychee) in a venv first on PATH - Ubuntu 24.04 ships PEP 668, so there is no system interpreter to install into. Everything it installs by name is pinned: pip with ==, paho to a commit rather than a mutable tag, lychee to a per-architecture SHA-256 (the amd64 tarball is not the arm64 one, so a single literal would leave one leg unbuildable). Only UBUNTU_BASE_TAG reaches CI as a build arg, because it is the one value the variant tag can name; the rest are ARG defaults, the regime images/versions.json's _readme describes and check-pins.sh enforces. Two tools are wired up rather than merely installed, because installed is not the same as in the path of a build. /usr/lib/ccache ships gcc and g++ wrappers but no cc or c++ - the names CMake looks for first - so a PATH entry would leave a CMake project compiling through /usr/bin/c++ with the cache untouched; CMAKE_{C,CXX}_COMPILER_LAUNCHER is read by CMake itself and holds whatever compiler the project picks. CMAKE_EXPORT_COMPILE_COMMANDS is what makes `clang-tidy -p build` resolve after an ordinary `cmake -S . -B build`, rather than after a re-configure with one more flag. The verification layer asserts rather than lists. The two clang tools have to report the numbers pip was told to install - the wheel and the wrapper are different things - jsonschema has to appear at its pinned version in the freeze, CMake has to have taken the ccache launcher, and smoke/ is configured, built and ctest-ed inside the build: it proves find_package(GTest) resolves, that GMock links, and that the paho which loads reports the version pinned beside its commit. clang-tidy is then run over that same source, because a wheel whose binary cannot find its resource directory answers --version perfectly and fails on the first real file. That build runs with CCACHE_DISABLE=1: warming the cache as root would leave /opt/ccache/tmp root-owned and every later `-u $(id -u)` run failing on it. Nothing in the file pipes a tool's output anywhere, because the image sets no SHELL and dash has no pipefail, so `gcc --version | head -1` would take head's exit status and leave a missing compiler green. The image runs as the invoking user: git safe.directory (a mounted checkout is owned by a uid with no passwd entry, and without it both `git ls-files` and `git describe` fail inside the container only), a HOME of the image's own rather than /tmp, which a caller may replace mid-run, and 1777 on both that and the ccache directory - an arbitrary uid has to write them, and the sticky bit keeps one uid from replacing another's files. host.yml is its own workflow - nothing in this repo is built FROM it, so it pushes on master alone, like platformio.yml. The prose that counted the workflows is corrected with them: two sentences in CLAUDE.md said "both workflows" where there are now three, and dependabot.yml's list of what it deliberately does not track gains this image's pins - ARG defaults, a commit ref and two literal checksums, none of which any ecosystem resolves. --- .github/dependabot.yml | 7 + .github/workflows/host.yml | 234 +++++++++++++++++++++ CLAUDE.md | 46 +++-- README.md | 25 ++- images/host/Dockerfile | 298 ++++++++++++++++++++++++++ images/host/README.md | 345 +++++++++++++++++++++++++++++++ images/host/smoke/CMakeLists.txt | 43 ++++ images/host/smoke/smoke.cpp | 52 +++++ images/versions.json | 22 +- 9 files changed, 1048 insertions(+), 24 deletions(-) create mode 100644 .github/workflows/host.yml create mode 100644 images/host/Dockerfile create mode 100644 images/host/README.md create mode 100644 images/host/smoke/CMakeLists.txt create mode 100644 images/host/smoke/smoke.cpp diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 0db42ca..8d21663 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -32,6 +32,13 @@ # ecosystem resolves. # - IDF_BASE_TAG, ESP_MATTER_VERSION, PIO_VERSION. These are chosen against # Espressif's own compatibility matrix, not against "newest". +# - Everything images/host/Dockerfile pins. Its pip installs are ARG defaults, +# which no ecosystem resolves; PAHO_REF is a commit fetched by a `git fetch` +# line; and LYCHEE_VERSION travels with two literal checksums that a bump has +# to recompute per architecture. All three are bumped by hand, and the QA +# versions deliberately track what the consuming project pins for the same +# tools rather than what is newest - the image agreeing with its consumer is +# the whole point of it. # # Note that a bump here edits a workflow file, which is in that workflow's own # `paths:` filter - so every Dependabot PR triggers a full paid rebuild, and diff --git a/.github/workflows/host.yml b/.github/workflows/host.yml new file mode 100644 index 0000000..7846631 --- /dev/null +++ b/.github/workflows/host.yml @@ -0,0 +1,234 @@ +# Host Docker Build Workflow +# +# Build Strategy: +# - Main repo master: Build + Push to GHCR +# - Main repo dev/PR: Build only (validation) - nothing is uploaded. Like +# platformio.yml and unlike esp-idf.yml, no other image is built FROM this one, +# so there is nothing for a pull request to push for. +# - workflow_dispatch: Build on any branch of the main repo, push only on master +# - Forks: nothing runs - the runner pools are not reachable there; use +# ./scripts/build.sh instead +# - Multi-platform: Platform matrix builds linux/amd64 and linux/arm64 in parallel, +# each on a runner of its own architecture (matrix.runner) - no QEMU. This image +# is the one a developer on Apple Silicon runs interactively, so an emulated +# arm64 leg would be the slow half of exactly the audience it exists for. +name: ๐Ÿณ Host Docker Image + +on: + push: + branches: [master, dev] + # Negations come last and are order-sensitive; `paths:` and `paths-ignore:` + # cannot both be used for one event. Without the exclusion a + # documentation-only commit rebuilds and republishes the image. + paths: + - '.github/workflows/host.yml' + - 'images/host/**' + - 'images/versions.json' + - 'scripts/versions-matrix.sh' + - 'scripts/check-versions.sh' + - '!images/**/*.md' + pull_request: + branches: [master, dev] + paths: + - '.github/workflows/host.yml' + - 'images/host/**' + - 'images/versions.json' + - 'scripts/versions-matrix.sh' + - 'scripts/check-versions.sh' + - '!images/**/*.md' + workflow_dispatch: + +env: + REGISTRY: ghcr.io + HOST_IMAGE_NAME: jethome-dev-host + # build-push-action v6+ uploads a build record as a workflow artifact by default. + # This repository is public, so that archive - full build log and metadata - would + # be downloadable by anyone off the run page, on every leg of every push. The + # summary itself is kept: it reports what was built without opening the log, and + # costs no storage. + DOCKER_BUILD_RECORD_UPLOAD: false + +# Superseded pull-request runs are cancelled - nothing has been published yet and +# they cost money. Pushes are deliberately NOT grouped together: the group key +# includes the commit, so each one gets its own group and none can cancel another. +# +# `cancel-in-progress: false` would not have given a queue. GitHub cancels a +# *pending* run in a group whenever a newer one arrives, regardless of that flag - +# so grouping master pushes by ref would silently drop the middle commit of any +# three that land inside one build window, and its sha- image, which the +# READMEs document as the way to pin an exact commit, would never exist. +concurrency: + group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name == 'pull_request' && 'pr' || github.sha }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +jobs: + # Both matrices come from images/versions.json, so a version lives in one place + # instead of two matrix blocks that have to be edited together. The consistency + # check runs first and fails the whole workflow. + prepare: + name: ๐Ÿงฎ Resolve matrices + runs-on: ubuntu-latest + timeout-minutes: 5 + if: github.repository_owner == 'jethome-iot' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) + permissions: + contents: read + outputs: + build: ${{ steps.matrices.outputs.build }} + manifest: ${{ steps.matrices.outputs.manifest }} + steps: + - name: ๐Ÿ“ฅ Checkout repository + uses: actions/checkout@v7 + + - name: โœ… Check version consistency + run: ./scripts/check-versions.sh + + - name: ๐Ÿงฎ Generate matrices + id: matrices + run: | + set -euo pipefail + # Assigned first, echoed second: `echo "k=$(cmd)"` takes echo's exit + # status, so a failing generator would write an empty matrix and leave + # this step green. + build=$(./scripts/versions-matrix.sh host build) + manifest=$(./scripts/versions-matrix.sh host manifest) + { + echo "build=${build}" + echo "manifest=${manifest}" + } >> "$GITHUB_OUTPUT" + + host-build: + name: host-build (${{ matrix.tag }}, ${{ matrix.platform }}) + needs: prepare + runs-on: ${{ matrix.runner }} + timeout-minutes: ${{ matrix.timeout_minutes }} + # Owner-only, with no workflow_dispatch escape hatch: the runner pools belong + # to jethome-iot, a fork cannot resolve their labels, and an unresolvable + # `runs-on` queues for 24 hours rather than failing. A fork that wants to build + # this image runs ./scripts/build.sh. + # Two conditions, not one. The owner check keeps this out of forks that run the + # workflow themselves. The head-repo check keeps it out of pull requests *from* + # a fork, where github.repository_owner is still jethome-iot - so without it a + # fork-controlled Dockerfile would execute on this org's paid runner pools. + if: github.repository_owner == 'jethome-iot' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) + permissions: + contents: read + packages: write + strategy: + fail-fast: false + matrix: + include: ${{ fromJSON(needs.prepare.outputs.build) }} + steps: + - name: ๐Ÿ“ฅ Checkout repository + uses: actions/checkout@v7 + + - name: ๐Ÿ”ง Set up Docker Buildx + uses: docker/setup-buildx-action@v4 + + - name: ๐Ÿ” Log in to GitHub Container Registry + if: github.ref_name == 'master' + uses: docker/login-action@v4 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Pushed by digest, with no tag of its own: a tag here would be a mutable + # name that the manifest job below has to look up again later, and between + # those two moments another run can overwrite it. The digest is the only + # thing handed forward. + - name: ๐Ÿณ Build and push by digest + id: build + uses: docker/build-push-action@v7 + with: + context: images/host + platforms: ${{ matrix.platform }} + outputs: type=image,name=${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.HOST_IMAGE_NAME }},push-by-digest=true,name-canonical=true,push=${{ github.ref_name == 'master' }} + build-args: ${{ matrix.build_args }} + + # Matrix legs cannot each set a job output - they would overwrite one + # another - so the digests travel as one empty file per leg, named after the + # digest itself. + - name: ๐Ÿ“ค Export digest + if: github.ref_name == 'master' + run: | + mkdir -p /tmp/digests + digest="${{ steps.build.outputs.digest }}" + touch "/tmp/digests/${digest#sha256:}" + + - name: โฌ†๏ธ Upload digest + if: github.ref_name == 'master' + uses: actions/upload-artifact@v7 + with: + name: digest-${{ env.HOST_IMAGE_NAME }}-${{ matrix.tag }}-${{ matrix.platform_tag }} + path: /tmp/digests/* + if-no-files-found: error + retention-days: 1 + + host-manifest: + name: host-manifest (${{ matrix.tag }}) + runs-on: ubuntu-latest + timeout-minutes: 10 + needs: [prepare, host-build] + # See the note in esp-idf.yml: one variant's failure must not withhold another + # variant's tags. + if: >- + ${{ !cancelled() + && needs.prepare.result == 'success' + && github.repository_owner == 'jethome-iot' + && github.ref_name == 'master' }} + permissions: + contents: read + packages: write + strategy: + fail-fast: false + matrix: + include: ${{ fromJSON(needs.prepare.outputs.manifest) }} + steps: + - name: ๐Ÿ” Log in to GitHub Container Registry + uses: docker/login-action@v4 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: โฌ‡๏ธ Download digests + uses: actions/download-artifact@v8 + with: + pattern: digest-${{ env.HOST_IMAGE_NAME }}-${{ matrix.tag }}-* + merge-multiple: true + path: /tmp/digests + + - name: ๐Ÿณ Create and push multi-arch manifest + env: + IMAGE: ${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.HOST_IMAGE_NAME }} + TAG: ${{ matrix.tag }} + PRIMARY: ${{ matrix.primary }} + EXPECTED_PLATFORMS: ${{ matrix.platform_count }} + run: | + set -euo pipefail + cd /tmp/digests + found=$(find . -type f | wc -l | tr -d ' ') + if [ "${found}" -ne "${EXPECTED_PLATFORMS}" ]; then + echo "::error::expected ${EXPECTED_PLATFORMS} platform digests, found ${found}" + exit 1 + fi + # Every source is a digest this run produced, so nothing here can be + # resolved to another run's image. Built as an array rather than an + # unquoted expansion so the refs survive as separate arguments without + # relying on word splitting. + refs=() + for digest in *; do + refs+=("${IMAGE}@sha256:${digest}") + done + # Only the primary variant moves `latest` and the bare `sha-`. + # Every variant gets `-sha-`: the version tag itself is + # rewritten on every push, so without it a consumer pinned to a + # non-primary variant has no name for the build they were running. + SHA_SHORT="${GITHUB_SHA:0:7}" + tags=() + if [ "${PRIMARY}" = "true" ]; then + tags+=(-t "${IMAGE}:latest" -t "${IMAGE}:sha-${SHA_SHORT}") + fi + tags+=(-t "${IMAGE}:${TAG}-sha-${SHA_SHORT}") + tags+=(-t "${IMAGE}:${TAG}") + docker buildx imagetools create "${tags[@]}" "${refs[@]}" diff --git a/CLAUDE.md b/CLAUDE.md index 1151de2..e3a9598 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,14 +4,15 @@ This repo ships no application code. It is a home for Docker developer images: each image lives in `images//` and is published as `ghcr.io//jethome-dev-`. The deliverables are Dockerfiles, GitHub Actions workflows and READMEs. `images/` is the roster โ€” today `esp-idf`, -`esp-matter` and `platformio`, and adding to it is the normal case, not an +`esp-matter`, `host` and `platformio`, and adding to it is the normal case, not an exception. Rules below are written per image; where they name one, it is an example. ## Layout - One directory per image: `images//`, holding its `Dockerfile`, `README.md` - and any support files (`esp-matter/entrypoint.sh`, `platformio/pio_project/`). + and any support files (`esp-matter/entrypoint.sh`, `platformio/pio_project/`, + `host/smoke/`). - Each workflow sets `context: images/` and `scripts/build.sh` builds the same directory, so a Dockerfile can only `COPY` from inside its own directory โ€” a shared file at the repo root breaks both CI and local builds. @@ -20,11 +21,13 @@ example. ## CI -The authoritative files are `.github/workflows/esp-idf.yml` and -`.github/workflows/platformio.yml` โ€” read them before changing anything here. +The authoritative files are `.github/workflows/esp-idf.yml`, +`.github/workflows/platformio.yml` and `.github/workflows/host.yml` โ€” read them +before changing anything here. -- One workflow file per image *family*; today there are two โ€” `esp-idf.yml` builds - **both** esp-idf and esp-matter, `platformio.yml` builds platformio. A family is: +- One workflow file per image *family*; today there are three โ€” `esp-idf.yml` + builds **both** esp-idf and esp-matter, `platformio.yml` builds platformio, + `host.yml` builds host. A family is: an image whose Dockerfile is `FROM` another image of this repo joins the base image's workflow, adds `images//**` to its push **and** pull_request `paths:` filters, and chains via `needs: -build` โ€” the *build* job, not the @@ -97,7 +100,7 @@ The authoritative files are `.github/workflows/esp-idf.yml` and runtime `runs-on` is invisible to actionlint); every `base_tag` exists among the base image's tags; and the base image's `ARG` default names the primary variant's base tag. Run it locally with `./scripts/check-versions.sh`. -- The checker validates the whole file and both workflows gate on it, so a version +- The checker validates the whole file and every build workflow gates on it, so a version file broken for one image blocks publishing for all of them. That is deliberate โ€” the data is shared โ€” but it is also why `images/versions.json` sits in every `paths:` filter: a bump to one image rebuilds the others. The alternative, a file @@ -118,7 +121,10 @@ The authoritative files are `.github/workflows/esp-idf.yml` and - An `ARG` with no matrix entry is a pin CI never passes, so its Dockerfile default is the sole source of truth and is bumped there โ€” today `ESP32_PLATFORM_VERSION`, `NATIVE_PLATFORM_VERSION` and `UNITY_VERSION` in - `images/platformio/Dockerfile`. + `images/platformio/Dockerfile`, and every tool pin in `images/host/Dockerfile` + (the QA versions, `PAHO_VERSION`/`PAHO_REF`, `LYCHEE_VERSION` and its two + checksums). host passes one arg and one only, `UBUNTU_BASE_TAG`, because that is + the single value its tag can name. - Every matrix carries `fail-fast: false`, so one platform leg failing does not cancel the other and truncate its log. - `concurrency` cancels superseded **pull-request** runs and groups nothing else: @@ -144,9 +150,9 @@ The authoritative files are `.github/workflows/esp-idf.yml` and - **Whether the image is uploaded** differs per image. `esp-idf-build` pushes by digest on *every* run โ€” its GHCR login is unconditional to match โ€” because `esp-matter-build` consumes that digest and that is what makes ESP-Matter - validatable on a pull request. `platformio-build` and `esp-matter-build` keep - `push=${{ github.ref_name == 'master' }}`: nothing consumes them, so a PR - builds and throws away. + validatable on a pull request. `platformio-build`, `host-build` and + `esp-matter-build` keep `push=${{ github.ref_name == 'master' }}`: nothing + consumes them, so a PR builds and throws away. The org literal is hardcoded throughout. - **Every image is build-validated on pull requests**, ESP-Matter included, and on @@ -165,7 +171,7 @@ The authoritative files are `.github/workflows/esp-idf.yml` and the owner check alone would let a fork-controlled Dockerfile run on this org's paid pools. The condition is `github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository`. -- Both workflows exclude `!images/**/*.md` from their `paths:`. Negations come last +- Every build workflow excludes `!images/**/*.md` from its `paths:`. Negations come last and are order-sensitive, and `paths:` cannot be mixed with `paths-ignore:` for one event. Without this a documentation-only commit rebuilds and republishes every image. @@ -245,7 +251,14 @@ The authoritative files are `.github/workflows/esp-idf.yml` and `/opt/esp/python-packages.txt`, a file rather than a pipe so nothing swallows a failure, and **an image that installs on top regenerates it**: esp-matter's `install.sh` populates the same venv, so inheriting the base's snapshot would - leave the file describing an environment that no longer exists. + leave the file describing an environment that no longer exists. host takes the + same rule one step further, because a version print cannot reach what it + promises: its layer asserts that `clang-format` and `clang-tidy` *report* the + numbers pip was told to install (the wrapper and the wheel are two different + things), and then configures, builds and `ctest`s `images/host/smoke/` โ€” two + files that prove `find_package(GTest)` resolves, that GMock links, and that the + paho which loads reports the version pinned beside its commit. Its own freeze + goes to `/opt/qa-packages.txt`. - **A version an image installs by name is a version this repository chose**, and `./scripts/check-pins.sh` enforces that per package manager, because the price of a pin differs per manager. `pip` takes `==` on every name โ€” pip here runs with no @@ -281,13 +294,16 @@ The authoritative files are `.github/workflows/esp-idf.yml` and - Docker's default `SHELL` is `/bin/sh`, so an image that sets none must keep its `RUN` layers POSIX โ€” check the image's own Dockerfile before reaching for a bash-ism. esp-idf and esp-matter set `SHELL ["/bin/bash", "-c"]` because their - layers call the bash builtin `source`; platformio does not. A trailing + layers call the bash builtin `source`; platformio and host do not โ€” host's + multi-line layers are POSIX `sh` on purpose (`set -eu`, `case`, no arrays). A trailing `CMD ["/bin/bash"]` sets the interactive shell, not the build shell. - Toolchain activation is per-image, not repo-wide. The ESP images source `${IDF_PATH}/export.sh` in every `RUN` that needs the toolchain (esp-idf additionally needs `export IDF_PATH_FORCE=1` in the same `RUN`; esp-matter also sources `${ESP_MATTER_PATH}/export.sh`), while platformio puts `pio` on `PATH` - at install time and activates nothing. Either way call tools by name, never by + at install time and activates nothing, and host puts its venv (`/opt/qa-venv`) + first on `PATH` so `ruff`/`mypy`/`clang-tidy` resolve without activation โ€” + Ubuntu 24.04 ships PEP 668, so there is no system interpreter to install into. Either way call tools by name, never by absolute path: `/opt/esp/python_env/idf5.4_py3.12_env/bin/python3` was tried and reverted โ€” it pins a version-stamped directory that a version bump invalidates. - esp-matter activates both environments at runtime through diff --git a/README.md b/README.md index 4a339fb..49da9bd 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,7 @@ Docker-based development environment for embedded systems, providing containeriz [![ESP-IDF Docker Image](https://github.com/jethome-iot/jethome-dev/actions/workflows/esp-idf.yml/badge.svg?branch=master)](https://github.com/jethome-iot/jethome-dev/actions/workflows/esp-idf.yml) [![PlatformIO Docker Image](https://github.com/jethome-iot/jethome-dev/actions/workflows/platformio.yml/badge.svg?branch=master)](https://github.com/jethome-iot/jethome-dev/actions/workflows/platformio.yml) +[![Host Docker Image](https://github.com/jethome-iot/jethome-dev/actions/workflows/host.yml/badge.svg?branch=master)](https://github.com/jethome-iot/jethome-dev/actions/workflows/host.yml) ## Current Images @@ -11,6 +12,7 @@ Docker-based development environment for embedded systems, providing containeriz |-------|-------------|---------------| | [esp-idf](./images/esp-idf/) | ESP-IDF for every ESP32 chip, plus pytest and QEMU emulation for some of them | [README](./images/esp-idf/README.md) | | [esp-matter](./images/esp-matter/) | ESP-Matter SDK for Matter protocol development on ESP32 | [README](./images/esp-matter/README.md) | +| [host](./images/host/) | Host (POSIX) builds and QA: GCC, CMake, Ninja, GTest/GMock, paho.mqtt.c, clang-tidy, ruff, lychee | [README](./images/host/README.md) | | [platformio](./images/platformio/) | PlatformIO with ESP32 platform support + ESP-IDF + Unity testing | [README](./images/platformio/README.md) | ## Quick Start @@ -29,9 +31,9 @@ docker run -it --rm -v $(pwd):/workspace \ ``` The build command differs per image (`idf.py` for esp-idf and esp-matter, `pio` -for platformio). What each image contains, its supported chips, available tags, -build arguments and ready-to-run examples are documented in the image README -linked in the table above. +for platformio, a plain `cmake` invocation for host). What each image contains, +its supported chips, available tags, build arguments and ready-to-run examples are +documented in the image README linked in the table above. ## Local Development @@ -45,7 +47,7 @@ linked in the table above. # Build specific image (tagged as 'local') ./scripts/build.sh esp-idf -./scripts/build.sh platformio +./scripts/build.sh host # Build and run image interactively ./scripts/build.sh -r esp-idf @@ -122,7 +124,8 @@ It never runs on a pull request. # change touches that workflow's paths: # esp-idf.yml -> .github/workflows/esp-idf.yml, images/esp-idf/**, images/esp-matter/** # platformio.yml -> .github/workflows/platformio.yml, images/platformio/** -# Both workflows also support workflow_dispatch (Actions tab), which ignores the +# host.yml -> .github/workflows/host.yml, images/host/** +# Every workflow also supports workflow_dispatch (Actions tab), which ignores the # path filters. Images are pushed to GHCR from master only. # # In a fork nothing runs at all: the build jobs require the jethome-iot owner, @@ -158,11 +161,11 @@ This works because the **ESP-IDF** build job pushes by digest on every run โ€” i the only image another one is built from, so it is the only one that needs to. Those pushes carry no tag, and only `master` ever writes `latest` or a version tag. The untagged blobs a pull request leaves in GHCR accumulate and want an occasional -cleanup. PlatformIO, which nothing builds on, still uploads nothing outside +cleanup. PlatformIO and Host, which nothing builds on, upload nothing outside `master`. -Documentation-only changes trigger nothing: `!images/**/*.md` is excluded from both -workflows' path filters. +Documentation-only changes trigger nothing: `!images/**/*.md` is excluded from +every image workflow's path filters. ### Manual Building @@ -182,6 +185,7 @@ jethome-dev/ โ”‚ โ””โ”€โ”€ workflows/ # GitHub Actions workflows โ”‚ โ”œโ”€โ”€ esp-idf.yml # ESP-IDF and ESP-Matter image workflows โ”‚ โ”œโ”€โ”€ platformio.yml # PlatformIO image workflow +โ”‚ โ”œโ”€โ”€ host.yml # Host (POSIX) image workflow โ”‚ โ”œโ”€โ”€ lint.yml # actionlint + shellcheck (gates), hadolint (advisory) โ”‚ โ””โ”€โ”€ runner-smoke.yml # Reports what each runner pool actually is โ”œโ”€โ”€ images/ @@ -193,6 +197,10 @@ jethome-dev/ โ”‚ โ”‚ โ”œโ”€โ”€ Dockerfile # Image definition โ”‚ โ”‚ โ”œโ”€โ”€ entrypoint.sh # Activates ESP-IDF + ESP-Matter env on start โ”‚ โ”‚ โ””โ”€โ”€ README.md # Detailed documentation +โ”‚ โ”œโ”€โ”€ host/ # Host (POSIX) build, test and QA image +โ”‚ โ”‚ โ”œโ”€โ”€ Dockerfile # Image definition +โ”‚ โ”‚ โ”œโ”€โ”€ README.md # Detailed documentation +โ”‚ โ”‚ โ””โ”€โ”€ smoke/ # CMake project the verification layer builds and runs โ”‚ โ””โ”€โ”€ platformio/ # PlatformIO development image โ”‚ โ”œโ”€โ”€ Dockerfile # Image definition โ”‚ โ”œโ”€โ”€ README.md # Detailed documentation @@ -202,6 +210,7 @@ jethome-dev/ โ”‚ โ”œโ”€โ”€ lint.sh # Runs the same linters as CI, locally โ”‚ โ”œโ”€โ”€ versions-matrix.sh # Turns versions.json into the CI matrices โ”‚ โ”œโ”€โ”€ check-versions.sh # Enforces versions.json against the Dockerfiles +โ”‚ โ”œโ”€โ”€ check-pins.sh # Enforces that what the images install is pinned โ”‚ โ””โ”€โ”€ update-matter-ref.sh # Reports/advances the pinned ESP-Matter commits โ”œโ”€โ”€ CLAUDE.md # Repository conventions, loaded by Claude Code โ”œโ”€โ”€ LICENSE diff --git a/images/host/Dockerfile b/images/host/Dockerfile new file mode 100644 index 0000000..e519424 --- /dev/null +++ b/images/host/Dockerfile @@ -0,0 +1,298 @@ +# JetHome host development image +# +# Builds and tests the POSIX target of a C++ firmware project on the machine it +# runs on, and carries the QA gates that project runs beside the build: ruff, +# mypy, pytest, clang-format, clang-tidy and lychee. +# +# Why an image rather than a list of `apt-get install` lines in a CI job: a static +# analyzer reads the *host's* standard library, not the project's. libc++ (macOS) +# inlines the `throw` inside `std::string` and `std::function`; libstdc++ (Linux, +# and every CI runner) hides them behind external `__throw_*` the analyzer cannot +# look into. Same pinned clang-tidy, different findings - so "clang-tidy is green" +# means one thing on a developer's laptop and another in CI. This image is the +# second one, reproducible on either machine. +# +# Ubuntu, and the same release the CI runners use, for exactly that reason. Built +# for linux/amd64 AND linux/arm64 - an Apple Silicon developer must not land in +# QEMU, where the analysis they are waiting on takes an order of magnitude longer. +# +# The base tag is an ARG because images/versions.json requires every value it +# passes in `args` to appear in the variant's tag - that is what binds a published +# tag to the base it advertises. The tool versions below are deliberately NOT args: +# a tool version can never appear in the tag, so the same rule would reject it. +# They live as ARG defaults here, which is the regime images/versions.json's +# _readme describes for exactly this case, and ./scripts/check-pins.sh is what +# holds them to an exact version. +ARG UBUNTU_BASE_TAG=24.04 + +FROM ubuntu:${UBUNTU_BASE_TAG} + +# An ARG, not an ENV: this answers apt during the build and has no business +# reaching a shell the user opens later. +ARG DEBIAN_FRONTEND=noninteractive + +# โ”€โ”€ apt: the build, the host tests, and the tools the project's scripts call โ”€โ”€ +# +# GTest and GMock arrive as libraries and headers already built (Ubuntu ships +# .a archives plus the CMake package config), so a project consuming this image +# can `find_package(GTest)` and link - the verification layer at the bottom +# proves it rather than assuming it. +# +# The sanitizer runtimes are not listed: libasan, libubsan and libtsan come with +# g++ itself (/usr/lib/gcc//13/), so `-fsanitize=address` works out of the +# box and naming them here would only pin what build-essential already decides. +# +# git is not incidental either - a checkout mounted into this image is asked for +# `git ls-files` (which files a format gate covers) and `git describe` (the +# firmware version), so a build without it fails in a way that looks like a +# project bug. +RUN apt-get update && apt-get install -y --no-install-recommends \ + build-essential \ + cmake \ + ninja-build \ + ccache \ + pkg-config \ + libgtest-dev \ + libgmock-dev \ + git \ + ca-certificates \ + curl \ + jq \ + python3 \ + python3-venv \ + && apt-get autoremove -y \ + && apt-get clean \ + && rm -rf /var/lib/apt/lists/* + +# โ”€โ”€ pip: the QA toolchain, in a venv, pinned โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ +# +# A venv rather than the system interpreter because Ubuntu 24.04 ships PEP 668: +# a plain `pip install` into /usr refuses with externally-managed-environment. +# Putting the venv first on PATH is what keeps `ruff`, `mypy` and `clang-tidy` +# callable by bare name, and makes `python3 -m ruff` - the form a runner script +# uses when it wants the interpreter's own copy - resolve to this one. +# +# The numbers are this repository's copy of the versions the consuming project +# pins for the same tools. They are repeated rather than fetched: an image builds +# from its own tree, and nothing here can read another repository at build time. +# The LABELs at the bottom are what let the consumer verify the two agree instead +# of assuming it. +ARG RUFF_VERSION=0.15.20 +ARG MYPY_VERSION=2.1.0 +ARG PYTEST_VERSION=9.1.1 +ARG JSONSCHEMA_VERSION=4.26.0 +ARG CLANG_FORMAT_VERSION=22.1.5 +ARG CLANG_TIDY_VERSION=22.1.0 +ENV VIRTUAL_ENV=/opt/qa-venv +ENV PATH="${VIRTUAL_ENV}/bin:${PATH}" +RUN python3 -m venv "${VIRTUAL_ENV}" \ + && pip install --no-cache-dir \ + "ruff==${RUFF_VERSION}" \ + "mypy==${MYPY_VERSION}" \ + "pytest==${PYTEST_VERSION}" \ + "jsonschema==${JSONSCHEMA_VERSION}" \ + "clang-format==${CLANG_FORMAT_VERSION}" \ + "clang-tidy==${CLANG_TIDY_VERSION}" \ + && pip check + +# โ”€โ”€ paho.mqtt.c, from the commit the consuming project pins โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ +# +# The COMMIT, not the tag: a tag is a mutable ref, while the SHA is what makes the +# checkout reproducible - the same rule this repository already applies to the +# ESP-Matter pin. Git verifies the object against its hash, so no separate content +# check is needed. +# +# The distro package is not used, and the CMake flags are the ones the project's +# own CI action used: SSL off, because the framework links paho-mqtt3a alone and +# has no TLS backend on POSIX yet, and HIGH_PERFORMANCE on, which is paho's +# documented production mode. Building it any other way would make a test that +# passes here mean nothing about CI. +ARG PAHO_VERSION=v1.3.16 +ARG PAHO_REF=4a939ddb01eea581a32fd6f0adcfee51b91d2601 +ARG PAHO_REPO=https://github.com/eclipse-paho/paho.mqtt.c.git +RUN set -eu; \ + src="$(mktemp -d)"; \ + git -C "$src" init -q; \ + git -C "$src" fetch -q --depth 1 "${PAHO_REPO}" "${PAHO_REF}"; \ + git -C "$src" checkout -q FETCH_HEAD; \ + cmake -S "$src" -B "$src/build" -G Ninja \ + -DCMAKE_BUILD_TYPE=RelWithDebInfo \ + -DPAHO_BUILD_SHARED=TRUE \ + -DPAHO_WITH_SSL=FALSE \ + -DPAHO_HIGH_PERFORMANCE=TRUE \ + -DPAHO_ENABLE_TESTING=FALSE; \ + cmake --build "$src/build"; \ + cmake --install "$src/build"; \ + ldconfig; \ + rm -rf "$src" + +# โ”€โ”€ lychee, the offline markdown link checker โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ +# +# A release binary with a literal checksum per architecture: the amd64 tarball is +# not the arm64 one, so a single SHA would leave one of the two legs unbuildable, +# and fetching the .sha256 from the same release would verify transfer integrity +# only - an upstream compromise rotates both files in one act. A version bump +# therefore edits all three values by hand, deliberately outside Dependabot, which +# sees neither a `curl` argument nor an ARG. +ARG LYCHEE_VERSION=v0.24.2 +ARG LYCHEE_SHA256_AMD64=1f4e0ef7f6554a6ed33dd7ac144fb2e1bbed98598e7af973042fc5cd43951c9a +ARG LYCHEE_SHA256_ARM64=91a7bd65685da41b90ccb9bc867a3d649a7818042dae04ff405e55a25bddee4c +RUN set -eu; \ + case "$(dpkg --print-architecture)" in \ + amd64) triple=x86_64-unknown-linux-gnu; sha="${LYCHEE_SHA256_AMD64}" ;; \ + arm64) triple=aarch64-unknown-linux-gnu; sha="${LYCHEE_SHA256_ARM64}" ;; \ + *) echo "no lychee release for $(dpkg --print-architecture)" >&2; exit 1 ;; \ + esac; \ + art="lychee-${triple}.tar.gz"; \ + curl -sfL "https://github.com/lycheeverse/lychee/releases/download/lychee-${LYCHEE_VERSION}/${art}" -o "$art"; \ +# The one pipe in this file, and a safe one: echo cannot fail, so the pipeline's +# status is sha256sum's - which is what dash gives without `set -o pipefail`, an +# option it does not have. + echo "${sha} ${art}" | sha256sum -c; \ + tar -xz --strip-components=1 -f "$art" "lychee-${triple}/lychee"; \ + install -m 0755 lychee /usr/local/bin/lychee; \ + rm -f "$art" lychee + +# โ”€โ”€ running as the invoking user โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ +# +# The expected invocation mounts a checkout and passes `-u $(id -u):$(id -g)`, so +# the repository is owned by a uid this image knows nothing about. Without +# safe.directory git refuses it as "dubious ownership" and both `git ls-files` and +# `git describe` fail - inside the container only, which is the confusing kind of +# failure. It stays `'*'` rather than naming /workspace: safe.directory matches a +# worktree path exactly, and a consumer running this image as a job `container:` +# gets its checkout under /__w//, which a narrowed entry would refuse. +# +# HOME is a directory of this image's own, not /tmp: that uid has no passwd entry, +# so tools would otherwise write to /, and /tmp is scratch space a caller is +# entitled to replace - `--tmpfs /tmp`, a volume mounted over it, an `rm -rf +# /tmp/*` step - which would take HOME with it mid-run. Both it and the ccache +# directory are 1777 rather than 0777: an arbitrary uid has to be able to write +# them, and the sticky bit is what keeps one uid from replacing another's files +# once it can. +RUN git config --system --add safe.directory '*' +ENV HOME=/home/build +ENV CCACHE_DIR=/opt/ccache +RUN mkdir -p "${HOME}" "${CCACHE_DIR}" && chmod 1777 "${HOME}" "${CCACHE_DIR}" + +# โ”€โ”€ the two defaults that make the tools above actually apply โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ +# +# Installing ccache does not put it in the path of a build: /usr/lib/ccache ships +# `gcc` and `g++` wrappers but no `cc` or `c++`, which are the names CMake looks +# for first, so a PATH entry would leave a CMake project compiling straight +# through /usr/bin/c++ with the cache untouched. The launcher variables are read +# by CMake itself and hold whatever compiler the project picks; override with +# `-DCMAKE_CXX_COMPILER_LAUNCHER=` to build without the cache. +# +# The compile database is what clang-tidy resolves `-p ` against, and +# CMake writes it only when asked. Setting it here is what makes the analyzer this +# image exists for work against an ordinary `cmake -S . -B build` rather than +# against a re-configure with one more flag nobody remembers. +ENV CMAKE_C_COMPILER_LAUNCHER=ccache +ENV CMAKE_CXX_COMPILER_LAUNCHER=ccache +ENV CMAKE_EXPORT_COMPILE_COMMANDS=ON + +# โ”€โ”€ the pins, machine-readable โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ +# +# A consumer pins its own tool versions in its own tree; these labels are what let +# it assert the image agrees, so a drift between the two repositories surfaces as +# a named test failure rather than as a gate that quietly started meaning +# something else. +LABEL org.opencontainers.image.source="https://github.com/jethome-iot/jethome-dev" \ + org.opencontainers.image.description="Host (POSIX) build, test and QA image: GCC, CMake, Ninja, GTest/GMock, paho.mqtt.c, clang-format, clang-tidy, ruff, mypy, pytest, lychee" \ + dev.jethome.paho.version="${PAHO_VERSION}" \ + dev.jethome.paho.ref="${PAHO_REF}" \ + dev.jethome.clang-tidy.version="${CLANG_TIDY_VERSION}" \ + dev.jethome.clang-format.version="${CLANG_FORMAT_VERSION}" \ + dev.jethome.ruff.version="${RUFF_VERSION}" \ + dev.jethome.mypy.version="${MYPY_VERSION}" \ + dev.jethome.pytest.version="${PYTEST_VERSION}" \ + dev.jethome.jsonschema.version="${JSONSCHEMA_VERSION}" \ + dev.jethome.lychee.version="${LYCHEE_VERSION}" + +WORKDIR /workspace + +# โ”€โ”€ verification โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ +# +# Every line below is an assertion, not a listing. A tool that never installed +# still leaves a green build until something asks it for its version, and a +# library that installed as a stub still links until something calls it. +# +# smoke/ is a two-file CMake project that the layer configures, builds with Ninja +# and runs under ctest. It covers what a version print cannot: that +# `find_package(GTest)` resolves, that GMock links, that libpaho-mqtt3a is found +# by name, and that the paho actually loaded reports the version this Dockerfile +# pins - the pin and the artifact disagreeing fails the build here rather than +# shipping an image that contradicts its own label. Its build tree is deleted; the +# sources stay at /opt/smoke-src, so the same check can be re-run against a +# published image (the README says how). +# +# The two clang tools assert their number instead of printing it: what pip +# installed and what the wrapper executes are two different things, and "some +# clang-tidy" is precisely the state this image exists to rule out. clang-tidy is +# then *run*, over smoke.cpp and the compile database the build just wrote, +# because a wheel whose binary cannot find its own resource directory answers +# `--version` perfectly and then fails on the first real file - the same "some +# clang-tidy", one step further along. The ccache assertion is of the same kind: +# it checks that CMake took the launcher, since a cache nothing is wired to is +# indistinguishable from a working one until someone measures a rebuild. +# +# The freeze goes to a file rather than a pipe, so the image carries its own +# snapshot and nothing can swallow a failure - and jsonschema is asserted from it +# because it is the one pin no tool here prints a version for. +# +# Nothing below pipes a tool's output anywhere, and that is deliberate rather than +# tidy: this image sets no SHELL, so these layers run under dash, which has no +# `set -o pipefail`. `gcc --version | head -1` would therefore take head's exit +# status - a missing compiler would print an error and leave the layer green, +# which is the exact failure this layer exists to catch. The assertions compare +# with `case` for the same reason. +COPY smoke/ /opt/smoke-src/ +RUN set -eu; \ +# The build below runs as root, and ccache creates its subdirectories with the +# creating user's ownership and umask - so a cache warmed here would leave +# /opt/ccache/tmp root-owned and every later `-u $(id -u)` run failing with +# "failed to create temporary file ... Permission denied". Disabling the cache for +# this layer keeps the directory as `mkdir` left it: empty and 1777. The launcher +# is still exercised, since ccache is what CMake invokes either way. + CCACHE_DISABLE=1; export CCACHE_DISABLE; \ + cmake --version; \ + ninja --version; \ + gcc --version; \ + git --version; \ + jq --version; \ + ccache --version; \ + python3 --version; \ + ruff --version; \ + mypy --version; \ + pytest --version; \ + lychee --version; \ + clang_format_reported="$(clang-format --version)"; \ + clang_tidy_reported="$(clang-tidy --version)"; \ + echo "${clang_format_reported}"; \ + echo "${clang_tidy_reported}"; \ + case "${clang_format_reported}" in \ + *"${CLANG_FORMAT_VERSION}"*) ;; \ + *) echo "clang-format reports something other than ${CLANG_FORMAT_VERSION}" >&2; exit 1 ;; \ + esac; \ + case "${clang_tidy_reported}" in \ + *"${CLANG_TIDY_VERSION}"*) ;; \ + *) echo "clang-tidy reports something other than ${CLANG_TIDY_VERSION}" >&2; exit 1 ;; \ + esac; \ + pip freeze > /opt/qa-packages.txt; \ + grep -qx "clang-tidy==${CLANG_TIDY_VERSION}" /opt/qa-packages.txt; \ + grep -qx "clang-format==${CLANG_FORMAT_VERSION}" /opt/qa-packages.txt; \ + grep -qx "jsonschema==${JSONSCHEMA_VERSION}" /opt/qa-packages.txt; \ + cmake -S /opt/smoke-src -B /tmp/smoke-build -G Ninja \ + -DPAHO_EXPECTED_VERSION="${PAHO_VERSION}"; \ + grep -qx "CMAKE_CXX_COMPILER_LAUNCHER:STRING=ccache" /tmp/smoke-build/CMakeCache.txt; \ + cmake --build /tmp/smoke-build; \ + ctest --test-dir /tmp/smoke-build --output-on-failure; \ +# Real analysis over a real file: `--checks=-*` alone exits 0 whatever happens, +# so the check set is what makes a compilation error - a missing resource +# directory, an unusable system header - fail this layer. + clang-tidy --quiet "--checks=-*,bugprone-*" -p /tmp/smoke-build /opt/smoke-src/smoke.cpp; \ + rm -rf /tmp/smoke-build; \ + echo "jethome host image verified" + +CMD ["/bin/bash"] diff --git a/images/host/README.md b/images/host/README.md new file mode 100644 index 0000000..9d422e4 --- /dev/null +++ b/images/host/README.md @@ -0,0 +1,345 @@ +# Host Development Image + +Docker image for building, testing and QA-checking C++ projects **on the machine +that runs them** โ€” no cross-compiler, no target hardware. Ubuntu with GCC, CMake, +Ninja, GTest/GMock and a from-source [Eclipse Paho MQTT C](https://github.com/eclipse-paho/paho.mqtt.c) +client, plus the linters and analyzers a CI pipeline runs beside a build. + +## Overview + +A static analyzer reads the *host's* standard library, not the project's, so +`clang-tidy` means one thing under libc++ (macOS) and another under libstdc++ +(Linux, and every CI runner): the same pinned analyzer reports findings on one and +stays silent on the other. This image is the Linux answer, reproducible on any +machine โ€” a developer gets the CI verdict locally, on their own architecture, +without waiting for a pull-request run to tell them. + +Everything it installs by name is pinned: the Python tools with `==`, the Paho +client to a commit rather than a tag, and the `lychee` binary to a per-architecture +SHA-256. The versions are recorded as image labels, so a consumer can assert the +image agrees with its own pins instead of assuming it (see +[Verifying the pins](#verifying-the-pins)). + +## What's Inside + +**Base Environment:** +- Ubuntu (the release the tag names), matching the CI runner image +- Python 3 from the distribution, with the QA tools in a virtualenv at + `/opt/qa-venv` that is first on `PATH` + +**Build Tools:** +- build-essential (gcc, g++, make) โ€” the sanitizer runtimes (`libasan`, + `libubsan`, `libtsan`) come with it, so `-fsanitize=โ€ฆ` builds need nothing extra +- cmake, ninja-build, pkg-config +- ccache, wired into CMake builds through `CMAKE_{C,CXX}_COMPILER_LAUNCHER`, with + its cache at `/opt/ccache` + +**Libraries:** +- GTest and GMock as headers plus static archives and a CMake package config, so + `find_package(GTest REQUIRED)` resolves +- paho.mqtt.c, built from a pinned commit and installed into `/usr/local` + (`-DPAHO_WITH_SSL=FALSE -DPAHO_HIGH_PERFORMANCE=TRUE`, shared library) + +**QA Tools:** +- clang-format, clang-tidy โ€” same LLVM line, both pinned +- ruff, mypy, pytest, jsonschema +- lychee โ€” offline markdown link checker +- git, curl, jq + +## Quick Start + +### Available Tags + +| Tag Type | Example | Usage | +|----------|---------|-------| +| **Latest** | `latest` | Always points to newest build (floating) | +| **Version** | `ubuntu-` | Pin to a specific Ubuntu base (recommended for CI/CD) | +| **Commit** | `sha-` | Pin to exact git commit (debugging); the commit is the first 7 characters, e.g. `sha-9c281e3` | +| **Version + commit** | `ubuntu--sha-` | Immutable: the only tag that never moves | + +**Tag Recommendations:** +- **Development**: `latest` for convenience +- **CI/CD**: the version tag for reproducibility +- **Rolling back**: `ubuntu--sha-` โ€” the version tag itself is + rewritten on every rebuild + +### Pull Image + +```bash +# Latest build +docker pull ghcr.io/jethome-iot/jethome-dev-host:latest + +# Specific Ubuntu base (recommended for CI/CD) +docker pull ghcr.io/jethome-iot/jethome-dev-host:ubuntu- +``` + +### Build a Project + +```bash +docker run --rm \ + -u $(id -u):$(id -g) \ + -v $(pwd):/workspace \ + ghcr.io/jethome-iot/jethome-dev-host:latest \ + bash -c 'cmake -S . -B build -G Ninja && cmake --build build' +``` + +### Run Tests + +```bash +docker run --rm \ + -u $(id -u):$(id -g) \ + -v $(pwd):/workspace \ + ghcr.io/jethome-iot/jethome-dev-host:latest \ + ctest --test-dir build --output-on-failure +``` + +### Run the Analyzers + +```bash +# Formatting, over the files git knows about +docker run --rm -u $(id -u):$(id -g) -v $(pwd):/workspace \ + ghcr.io/jethome-iot/jethome-dev-host:latest \ + bash -c 'git ls-files "*.cpp" "*.h" | xargs clang-format --dry-run --Werror' + +# Static analysis, against the compile database the build above produced. The +# image sets CMAKE_EXPORT_COMPILE_COMMANDS=ON, so an ordinary `cmake -S . -B build` +# writes build/compile_commands.json and `-p build` resolves without a second flag. +# The check set comes from the project's own .clang-tidy; with no such file +# clang-tidy has nothing enabled and exits non-zero, so name the checks instead +docker run --rm -u $(id -u):$(id -g) -v $(pwd):/workspace \ + ghcr.io/jethome-iot/jethome-dev-host:latest \ + clang-tidy -p build src/main.cpp # project with .clang-tidy + +docker run --rm -u $(id -u):$(id -g) -v $(pwd):/workspace \ + ghcr.io/jethome-iot/jethome-dev-host:latest \ + clang-tidy '--checks=-*,bugprone-*' -p build src/main.cpp # project without one + +# Markdown links. `--offline` skips the network and checks that relative links +# resolve to files on disk; `--include-fragments` adds the #anchor check, which is +# off by default. A project with a lychee.toml of its own needs neither flag +docker run --rm -u $(id -u):$(id -g) -v $(pwd):/workspace \ + ghcr.io/jethome-iot/jethome-dev-host:latest \ + lychee --offline --include-fragments '**/*.md' +``` + +### Interactive Shell + +```bash +docker run -it --rm \ + -v $(pwd):/workspace \ + ghcr.io/jethome-iot/jethome-dev-host:latest +``` + +## Running as the Invoking User + +Pass `-u $(id -u):$(id -g)` and build output in `/workspace` belongs to you rather +than to root. The image is set up for it: + +- `git config --system --add safe.directory '*'` โ€” a mounted checkout is owned by + a uid the image has no passwd entry for, and without this git refuses it as + *dubious ownership*, taking `git ls-files` and `git describe` with it. +- `HOME=/home/build` โ€” that uid has no home directory, and tools that write one + would otherwise try `/`. It is a directory of the image's own rather than `/tmp`, + which a caller is free to replace (`--tmpfs /tmp`, a volume over it, a cleanup + step) and would take `HOME` with it mid-run. +- `CCACHE_DIR=/opt/ccache`, mode `1777` โ€” mount a volume there to keep the cache + between runs. Two things to know before sharing one: + - **One uid per cache.** ccache creates its subdirectories with the creating + user's ownership and umask, so a cache first written by one uid and then used + by another fails with *failed to create temporary file โ€ฆ Permission denied*. + Same uid, or `-e CCACHE_UMASK=000` from the start. + - **One trust level per cache.** On a hit ccache returns the stored object file + without re-deriving it, so a cache shared between untrusted (pull-request) and + trusted (release) builds lets the first decide what the second links. + +On Docker Desktop (macOS, Windows) ownership is mapped for you, so the flag +changes nothing; on Linux it is what keeps `build/` writable afterwards. + +## Verifying the Pins + +The image records what it was built with as OCI labels, so a project that pins the +same tools can assert the two agree rather than trusting the tag: + +```bash +docker inspect --format '{{json .Config.Labels}}' \ + ghcr.io/jethome-iot/jethome-dev-host:latest | jq . +``` + +```text +dev.jethome.paho.version, dev.jethome.paho.ref # tag and the exact commit +dev.jethome.clang-tidy.version, โ€ฆclang-format.version +dev.jethome.ruff.version, โ€ฆmypy.version, โ€ฆpytest.version, โ€ฆjsonschema.version +dev.jethome.lychee.version +``` + +The Python environment carries its own snapshot at `/opt/qa-packages.txt` +(`pip freeze` as of the build), which includes the transitive dependencies the +labels do not name. + +## What It Does Not Carry + +- **No Docker client or daemon.** A test suite that stands up containers of its + own โ€” a broker, a database โ€” cannot do it from inside this image; run those legs + on the host, or give the container a socket and a client yourself. +- **No cross-compilers and no target SDKs.** Firmware targets are the job of the + other images in [the repository index](../../README.md#current-images). +- **No TLS in the MQTT client.** Paho is built with `PAHO_WITH_SSL=FALSE`, so + `paho-mqtt3a` and `paho-mqtt3c` are present and `paho-mqtt3as`/`paho-mqtt3cs` + are not. +- **The distribution's Python**, not a specific minor. If a project's own CI pins + an interpreter version, its `pytest` legs run here on a different one; linters + are unaffected where the target version is set in configuration + (`python_version` for mypy, `target-version` for ruff) rather than taken from + the interpreter. + +## Usage Examples + +### CI/CD Integration + +**GitHub Actions:** + +```yaml +name: Build and test + +on: [push, pull_request] + +jobs: + posix: + runs-on: ubuntu-latest + container: + image: ghcr.io/jethome-iot/jethome-dev-host:latest + + steps: + - uses: actions/checkout@v7 + + - name: Configure and build + run: | + cmake -S . -B build -G Ninja + cmake --build build + + - name: Test + run: ctest --test-dir build --output-on-failure + + - name: Lint + run: | + ruff check . + mypy +``` + +**GitLab CI:** + +```yaml +posix: + image: ghcr.io/jethome-iot/jethome-dev-host:latest + + script: + - cmake -S . -B build -G Ninja + - cmake --build build + - ctest --test-dir build --output-on-failure +``` + +### Sanitizer Builds + +```bash +docker run --rm -u $(id -u):$(id -g) -v $(pwd):/workspace \ + ghcr.io/jethome-iot/jethome-dev-host:latest \ + bash -c 'cmake -S . -B build-asan -G Ninja \ + -DCMAKE_BUILD_TYPE=Debug \ + -DCMAKE_CXX_FLAGS="-fsanitize=address,undefined" && \ + cmake --build build-asan' +``` + +## Environment Variables + +``` +VIRTUAL_ENV=/opt/qa-venv # the QA virtualenv, first on PATH +HOME=/home/build # the invoking uid has no home directory; 1777 +CCACHE_DIR=/opt/ccache # 1777; mount a volume to persist it +CMAKE_C_COMPILER_LAUNCHER=ccache # what actually puts ccache in the build +CMAKE_CXX_COMPILER_LAUNCHER=ccache # override with -DCMAKE_CXX_COMPILER_LAUNCHER= +CMAKE_EXPORT_COMPILE_COMMANDS=ON # so clang-tidy -p just works +``` + +Your project files live in `/workspace` (mount as volume). + +## Building the Image + +### Standard Build + +```bash +cd images/host +docker build -t jethome-dev-host:local . +``` + +Or from the repository root: `./scripts/build.sh host`. + +### Custom Build Arguments + +```bash +docker build \ + --build-arg UBUNTU_BASE_TAG= \ + --build-arg CLANG_TIDY_VERSION= \ + --build-arg PAHO_REF= \ + -t jethome-dev-host:local . +``` + +Available build arguments (defaults: see the Dockerfile): +- `UBUNTU_BASE_TAG` โ€” the Ubuntu base image tag; the only argument CI passes, and + the one the image tag names +- `RUFF_VERSION`, `MYPY_VERSION`, `PYTEST_VERSION`, `JSONSCHEMA_VERSION` โ€” the + Python QA tools +- `CLANG_FORMAT_VERSION`, `CLANG_TIDY_VERSION` โ€” the LLVM tools; the verification + layer asserts the installed binaries report these numbers +- `PAHO_VERSION`, `PAHO_REF`, `PAHO_REPO` โ€” the MQTT client's tag, exact commit and + origin. `PAHO_VERSION` is what the built library is checked against, so it and + `PAHO_REF` are bumped together +- `LYCHEE_VERSION`, `LYCHEE_SHA256_AMD64`, `LYCHEE_SHA256_ARM64` โ€” the link + checker's release and its per-architecture checksums; a version bump edits all + three + +The last layer of the build is a verification step, and it asserts rather than +lists: it prints every tool's version, requires the two clang tools to report the +pinned numbers and `jsonschema` to appear at its pinned version in the freeze, +then configures, builds and `ctest`s the small CMake project in +[`smoke/`](./smoke/) โ€” proving that `find_package(GTest)` resolves, that GMock +links, that CMake took the ccache launcher, and that the Paho library loaded +reports the pinned version. It finishes by running `clang-tidy` over that +project's own source against the compile database the build wrote, so an analyzer +that answers `--version` but cannot find its resource directory fails the image +instead of the user's first run. + +The sources stay in the image at `/opt/smoke-src`, so the same check runs against +a published image: + +```bash +IMAGE=ghcr.io/jethome-iot/jethome-dev-host:latest +# The version to check against comes from the image's own label, so this does not +# repeat a number that lives in the Dockerfile +PAHO=$(docker inspect --format '{{index .Config.Labels "dev.jethome.paho.version"}}' "$IMAGE") + +docker run --rm -e PAHO="$PAHO" "$IMAGE" bash -c ' + cmake -S /opt/smoke-src -B /tmp/smoke -G Ninja -DPAHO_EXPECTED_VERSION="$PAHO" >/dev/null && + cmake --build /tmp/smoke >/dev/null && + ctest --test-dir /tmp/smoke --output-on-failure' +``` + +### Multi-Platform Support + +This image is built for both **linux/amd64** and **linux/arm64**, each on a runner +of its own architecture. Docker pulls the right one for your machine โ€” an Apple +Silicon developer gets a native image rather than an emulated one. + +## Additional Resources + +- [GoogleTest](https://google.github.io/googletest/) +- [Eclipse Paho MQTT C](https://eclipse.dev/paho/index.php?page=clients/c/index.php) +- [clang-tidy](https://clang.llvm.org/extra/clang-tidy/) +- [lychee](https://lychee.cli.rs/) + +## License + +MIT License - see [LICENSE](../../LICENSE) file. + +## Related Images + +- [All images in this repository](../../README.md#current-images) diff --git a/images/host/smoke/CMakeLists.txt b/images/host/smoke/CMakeLists.txt new file mode 100644 index 0000000..55669d7 --- /dev/null +++ b/images/host/smoke/CMakeLists.txt @@ -0,0 +1,43 @@ +# Build-time smoke project for the image's verification layer. +# +# It is not a test suite for anything a user writes - it exists so the last layer +# of the Dockerfile can prove, rather than assume, that the pieces this image +# promises actually resolve, link and run: `find_package(GTest)` finding the +# distro package, GMock linking, libpaho-mqtt3a being findable by name, and the +# paho that loads reporting the version the Dockerfile pinned. +# +# It is compiled inside the image and its build tree is deleted afterwards. The +# sources are not: they stay at /opt/smoke-src, so the same check can be re-run +# against a published image without this repository at hand. +cmake_minimum_required(VERSION 3.22) +project(jethome_host_smoke CXX) + +set(CMAKE_CXX_STANDARD 17) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +# The Dockerfile passes its own ARG in, so the number is stated once, in the layer +# that installs paho. Required rather than defaulted: a smoke test that silently +# checks the empty string against itself is the failure it exists to catch. +if(NOT PAHO_EXPECTED_VERSION) + message(FATAL_ERROR "PAHO_EXPECTED_VERSION must be passed in (-DPAHO_EXPECTED_VERSION=v1.2.3)") +endif() +# The tag carries a leading `v`; the library reports the bare version. +string(REGEX REPLACE "^v" "" PAHO_EXPECTED_VERSION_BARE "${PAHO_EXPECTED_VERSION}") + +find_package(GTest REQUIRED) +find_library(PAHO_MQTT3A_LIBRARY NAMES paho-mqtt3a REQUIRED) +find_path(PAHO_MQTT3A_INCLUDE_DIR NAMES MQTTAsync.h REQUIRED) + +enable_testing() + +add_executable(smoke smoke.cpp) +target_include_directories(smoke PRIVATE "${PAHO_MQTT3A_INCLUDE_DIR}") +target_compile_definitions(smoke PRIVATE + PAHO_EXPECTED_VERSION="${PAHO_EXPECTED_VERSION_BARE}") +target_link_libraries(smoke PRIVATE + GTest::gtest + GTest::gtest_main + GTest::gmock + "${PAHO_MQTT3A_LIBRARY}") + +add_test(NAME smoke COMMAND smoke) diff --git a/images/host/smoke/smoke.cpp b/images/host/smoke/smoke.cpp new file mode 100644 index 0000000..3f454a3 --- /dev/null +++ b/images/host/smoke/smoke.cpp @@ -0,0 +1,52 @@ +// See CMakeLists.txt in this directory: compiled and run by the Dockerfile's +// verification layer, then deleted. Each test asserts something a `--version` +// print cannot reach. +#include + +#include +#include +#include + +namespace { + +class Clock { +public: + virtual ~Clock() = default; + virtual int now() const = 0; +}; + +class MockClock : public Clock { +public: + MOCK_METHOD(int, now, (), (const, override)); +}; + +// GMock is a separate library from GTest and a separate apt package; linking one +// says nothing about the other, and a project consuming this image needs both. +TEST(HostImageSmoke, GMockLinksAndMatches) { + MockClock clock; + EXPECT_CALL(clock, now()).WillOnce(::testing::Return(42)); + EXPECT_THAT(clock.now(), ::testing::Eq(42)); +} + +// Calls into libpaho-mqtt3a and checks what answered. A library that installed +// as a stub, or a second copy arriving from somewhere else, links just as well +// as the pinned one - the version it reports is what tells them apart, and it is +// compared against the commit this image was built from rather than merely +// printed. +TEST(HostImageSmoke, PahoLinksAndReportsThePinnedVersion) { + MQTTAsync_nameValue* info = MQTTAsync_getVersionInfo(); + ASSERT_NE(info, nullptr); + + const char* version = nullptr; + for (MQTTAsync_nameValue* entry = info; entry->name != nullptr; ++entry) { + if (std::strcmp(entry->name, "Version") == 0) { + version = entry->value; + break; + } + } + + ASSERT_NE(version, nullptr) << "paho reported no Version entry"; + EXPECT_STREQ(version, PAHO_EXPECTED_VERSION); +} + +} // namespace diff --git a/images/versions.json b/images/versions.json index e83b7d7..bd1818b 100644 --- a/images/versions.json +++ b/images/versions.json @@ -31,7 +31,11 @@ "survive in GHCR either way, they just stop being rebuilt.", "", "Not listed here: ARGs that CI never passes (ESP32_PLATFORM_VERSION,", - "NATIVE_PLATFORM_VERSION, UNITY_VERSION in images/platformio/Dockerfile). Their", + "NATIVE_PLATFORM_VERSION, UNITY_VERSION in images/platformio/Dockerfile, and", + "every tool pin in images/host/Dockerfile - the QA versions, PAHO_VERSION and", + "PAHO_REF, LYCHEE_VERSION and its two per-architecture checksums; host passes", + "UBUNTU_BASE_TAG and nothing else, because that is the one value its tag can", + "name). Their", "Dockerfile default is the only source of truth and is bumped there. The package", "pins inside a Dockerfile - the esp-idf test harness and esptool - are the same", "regime, and they could not live here even if that were desirable: every value in", @@ -109,6 +113,22 @@ } ] }, + "host": { + "platforms": { + "linux/amd64": "ubuntu-latest-8core", + "linux/arm64": "ubuntu-latest-8core-arm" + }, + "timeout_minutes": 60, + "builds": [ + { + "tag": "ubuntu-24.04", + "primary": true, + "args": { + "UBUNTU_BASE_TAG": "24.04" + } + } + ] + }, "platformio": { "platforms": { "linux/amd64": "ubuntu-latest-8core",