From 1e365e67d3b9325f183aa0c90073eeb764ef9972 Mon Sep 17 00:00:00 2001 From: Pavel Sokolov Date: Mon, 3 Aug 2026 18:37:00 +0300 Subject: [PATCH 1/5] Validate ESP-Matter on pull requests, and stop rebuilding on docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A pull request touching only images/esp-matter/** matched the workflow's paths filter, so esp-idf-build ran on its unchanged context and reported success. The check was not merely absent - it was misleading: a green "🐳 ESP-IDF Docker Image" on such a PR said nothing about whether ESP-Matter still compiled, and CLAUDE.md had to carry a paragraph warning readers not to trust it. The cause was `needs: esp-idf-manifest` on a job that only runs on master: needs on a skipped job skips the dependent too. The condition is now `${{ !cancelled() && … && needs.esp-idf-manifest.result != 'failure' }}`, so the job proceeds when that manifest was skipped but not when it failed. The base differs by context, deliberately. On master it is the digest artifact esp-idf-manifest just published - the exact-base guarantee stays intact. Elsewhere that artifact does not exist, so the build falls back to the last published idf- tag. That validates this image's own layers, which is what a change to it touches; a PR editing both images checks each against the other's released version, and the new combination is exercised when they land together. Worth being explicit about what this does not do: the workflows are not split into one file per image. Splitting would have broken the digest handoff, since the artifact is scoped to a single run - the alternative being a workflow_run trigger fetching another run's artifact through the API, or a return to building on a mutable tag. Both cost more than the duplicated esp-idf-build leg that a matter-only PR now runs, which is 1.5 minutes. Also excludes !images/**/*.md from both workflows' paths. The docs-only merge that started this whole series set off a full rebuild including a 5.5-hour ESP-Matter leg, and republished every tag for images whose contents had not changed. --- .github/workflows/esp-idf.yml | 72 ++++++++++++++++++++++---------- .github/workflows/platformio.yml | 6 +++ CLAUDE.md | 27 +++++++----- README.md | 23 +++++----- 4 files changed, 82 insertions(+), 46 deletions(-) diff --git a/.github/workflows/esp-idf.yml b/.github/workflows/esp-idf.yml index 32d97bd..37df23f 100644 --- a/.github/workflows/esp-idf.yml +++ b/.github/workflows/esp-idf.yml @@ -10,20 +10,22 @@ # - Multi-platform: Platform matrix builds linux/amd64 and linux/arm64 in parallel, # each on a runner of its own architecture (matrix.runner) - no QEMU # -# ESP-Matter exception: esp-matter-build needs esp-idf-manifest, which runs for -# jethome-iot on master only, so the ESP-Matter jobs are skipped (not -# build-validated) on dev, on pull requests, on workflow_dispatch outside master, -# and in forks entirely. The image is built FROM the esp-idf digest that manifest -# job publishes and hands over in the same run. -# Consequence: a change touching only images/esp-matter/** still matches the paths -# filter below, so esp-idf-build runs on its unchanged context and this workflow -# reports success - a green check here does not mean ESP-Matter compiles. -# Validate it with ./scripts/build.sh esp-matter, or on master. +# ESP-Matter base: on master it is the exact esp-idf digest esp-idf-manifest just +# published, handed over as an artifact within this run. On a pull request that job +# does not run, so esp-matter builds on the last published idf- tag +# instead - which validates its own layers, and is why a green check here does now +# mean ESP-Matter compiles. A PR editing both images checks each against the +# other's released version; the new combination is exercised on master. +# esp-matter-manifest stays master-only: publishing is. name: 🐳 ESP-IDF Docker Image on: push: branches: [master, dev] + # Negations come last and are order-sensitive; `paths:` and `paths-ignore:` + # cannot both be used for one event. Documentation-only commits used to + # trigger a full rebuild and republish every tag - the docs-only merge that + # started this series set off a 5.5-hour ESP-Matter leg. paths: - '.github/workflows/esp-idf.yml' - 'images/esp-idf/**' @@ -31,6 +33,7 @@ on: - 'images/versions.json' - 'scripts/versions-matrix.sh' - 'scripts/check-versions.sh' + - '!images/**/*.md' pull_request: branches: [master, dev] paths: @@ -40,6 +43,7 @@ on: - 'images/versions.json' - 'scripts/versions-matrix.sh' - 'scripts/check-versions.sh' + - '!images/**/*.md' workflow_dispatch: env: @@ -276,8 +280,12 @@ jobs: # leg skips the manifest job with it and leaves the published tags on the # previous build while the other platform's image sits in GHCR unreferenced. timeout-minutes: ${{ matrix.timeout_minutes }} - # Owner-only, as esp-idf-build; see the note there. - if: github.repository_owner == 'jethome-iot' + # esp-idf-manifest only runs on master, and `needs` on a skipped job would skip + # this one too - which is why ESP-Matter went unvalidated on every pull request + # while a green "🐳 ESP-IDF Docker Image" check implied otherwise. `!cancelled()` + # lets it proceed when that job was skipped, and the explicit failure test keeps + # it from building on a base whose publication actually failed. + if: ${{ !cancelled() && github.repository_owner == 'jethome-iot' && needs.esp-idf-manifest.result != 'failure' }} permissions: contents: read packages: write @@ -304,26 +312,44 @@ jobs: username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - # Named after the ESP-IDF version this image declares in its own tag, so the - # base it builds on and the version it advertises cannot drift apart: bump - # esp-idf without bumping the blocks below and this download fails outright - # instead of publishing an idf-v tag built on v. + # On master the base is the digest this very run published, downloaded from an + # artifact named after the ESP-IDF version this image declares in its own tag + # - so the base it builds on and the version it advertises cannot drift + # apart, and a half-done bump fails here instead of publishing an idf-v + # tag built on v. - name: ⬇️ Resolve base image digest + if: github.ref_name == 'master' uses: actions/download-artifact@v8 with: name: manifest-digest-${{ env.ESP_IDF_IMAGE_NAME }}-${{ matrix.base_tag }} path: /tmp/base - - name: 🏷️ Read base image digest + - name: 🏷️ Resolve base image reference id: base + env: + IMAGE: ${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.ESP_IDF_IMAGE_NAME }} + BASE_TAG: ${{ matrix.base_tag }} run: | set -euo pipefail - digest=$(cat /tmp/base/digest) - case "${digest}" in - sha256:*) ;; - *) echo "::error::base digest not usable: '${digest}'"; exit 1 ;; - esac - echo "digest=${digest}" >> "$GITHUB_OUTPUT" + if [ -f /tmp/base/digest ]; then + digest=$(cat /tmp/base/digest) + case "${digest}" in + sha256:*) ;; + *) echo "::error::base digest not usable: '${digest}'"; exit 1 ;; + esac + ref="${IMAGE}@${digest}" + else + # No artifact: this is a pull request, where esp-idf-manifest does not + # run and nothing has been published. Validating against the last + # published base is the point - it checks this image's own layers, + # which is what the change under review touches. A PR that also edits + # esp-idf is checked against that image's *previous* release; the + # combination is only exercised once both land on master. + ref="${IMAGE}:${BASE_TAG}" + echo "::notice::no digest artifact (not a master run) - validating against the published ${BASE_TAG}" + fi + echo "ref=${ref}" >> "$GITHUB_OUTPUT" + echo "base image: ${ref}" - name: 🐳 Build and push by digest id: build @@ -333,7 +359,7 @@ jobs: platforms: ${{ matrix.platform }} outputs: type=image,name=${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.ESP_MATTER_IMAGE_NAME }},push-by-digest=true,name-canonical=true,push=${{ github.ref_name == 'master' }} build-args: | - ${{ matrix.base_arg }}=${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.ESP_IDF_IMAGE_NAME }}@${{ steps.base.outputs.digest }} + ${{ matrix.base_arg }}=${{ steps.base.outputs.ref }} ${{ matrix.build_args }} - name: 📤 Export digest diff --git a/.github/workflows/platformio.yml b/.github/workflows/platformio.yml index 3663100..13eae19 100644 --- a/.github/workflows/platformio.yml +++ b/.github/workflows/platformio.yml @@ -13,12 +13,17 @@ name: 🐳 PlatformIO Docker Image on: push: branches: [master, dev] + # Negations come last and are order-sensitive; `paths:` and `paths-ignore:` + # cannot both be used for one event. Documentation-only commits used to + # trigger a full rebuild and republish every tag - the docs-only merge that + # started this series set off a 5.5-hour ESP-Matter leg. paths: - '.github/workflows/platformio.yml' - 'images/platformio/**' - 'images/versions.json' - 'scripts/versions-matrix.sh' - 'scripts/check-versions.sh' + - '!images/**/*.md' pull_request: branches: [master, dev] paths: @@ -27,6 +32,7 @@ on: - 'images/versions.json' - 'scripts/versions-matrix.sh' - 'scripts/check-versions.sh' + - '!images/**/*.md' workflow_dispatch: env: diff --git a/CLAUDE.md b/CLAUDE.md index 2168890..8324b3d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -123,16 +123,23 @@ The authoritative files are `.github/workflows/esp-idf.yml` and built and thrown away, never uploaded. Manifest jobs ask both in their own job `if`, since they have no build step to gate. The org literal is hardcoded. -- `esp-matter-build` has `needs: esp-idf-manifest`, which requires both - `jethome-iot` **and** `master`, so both ESP-Matter jobs are skipped entirely on - `dev`, on PRs and on non-master dispatch. The dependency is real: - `images/esp-matter/Dockerfile` is `FROM ${BASE_IMAGE}`, and the digest CI passes - there is published by that manifest job in the same run. -- Worse than "no validation": a PR touching only `images/esp-matter/**` still - matches the workflow's `paths:` filter, so `esp-idf-build` runs on its unchanged - context and the check goes green — a passing "🐳 ESP-IDF Docker Image" on such a - PR says nothing about ESP-Matter. Validate it with `./scripts/build.sh esp-matter` - or on `master`. +- **`esp-matter-build` runs on pull requests too**, and the base it uses differs by + context. On `master` it downloads the digest artifact `esp-idf-manifest` just + published and builds on that exact image. Elsewhere no such artifact exists — the + manifest job is master-only — so it falls back to the last published + `idf-` tag. That validates this image's own layers, which is what a + change to it touches; the *combination* of an edited esp-idf and an edited + esp-matter is only exercised once both land on master. +- That fallback is why the job's `if` is written as + `${{ !cancelled() && … && needs.esp-idf-manifest.result != 'failure' }}` rather + than a plain `needs:`. A `needs` on a skipped job skips this one as well, which + is exactly how ESP-Matter went unvalidated on every pull request while a green + "🐳 ESP-IDF Docker Image" check implied otherwise. Note the `${{ }}` wrapper — + YAML reserves `!` at the start of a scalar. +- Both workflows exclude `!images/**/*.md` from their `paths:`. Negations come last + and are order-sensitive, and `paths:` cannot be mixed with `paths-ignore:` for one + event. Without this a documentation-only commit rebuilds and republishes every + image. - Every build job runs on a larger-runner pool, chosen by platform through a `runner` key in the matrix `include:` and read as `runs-on: ${{ matrix.runner }}` — `ubuntu-latest-8core` for `linux/amd64`, `ubuntu-latest-8core-arm` for diff --git a/README.md b/README.md index 7023a38..f79ca5c 100644 --- a/README.md +++ b/README.md @@ -134,19 +134,16 @@ for your host architecture only and with the Dockerfile's own `ARG` defaults instead of the versions CI passes in, so a green local build is not a green CI run. -**Note:** an image built `FROM` another image of this repo is not build-validated on -`dev`. Its build job needs the base image's manifest job, which runs only for -`jethome-iot` on `master` — that job publishes the multi-arch tag the derived -Dockerfile pulls. Today that is ESP-Matter: `esp-matter-build` needs -`esp-idf-manifest`, so both ESP-Matter jobs are skipped on `dev`, on pull requests, -on `workflow_dispatch` outside `master`, and in forks entirely. Every other image -builds from its own context and is validated on `dev`. - -A change touching only `images/esp-matter/**` still matches the workflow's path -filter, so `esp-idf-build` runs on its unchanged context and the check reports -success: a green "🐳 ESP-IDF Docker Image" on such a PR does **not** mean the -ESP-Matter image compiles. Validate it locally with -`./scripts/build.sh esp-matter` (~50GB of disk, several hours) or on `master`. +**Note:** every image is build-validated on pull requests and on `dev`, including +one built `FROM` another image of this repo. What differs is the base it is built +on. On `master`, ESP-Matter builds on the exact ESP-IDF digest the same run just +published; anywhere else that digest does not exist yet, so it builds on the last +published `idf-` tag. A pull request therefore proves that the image +compiles, but a PR editing *both* images only proves each against the other's +released version — the new combination is exercised when they land on `master`. + +Documentation-only changes trigger nothing: `!images/**/*.md` is excluded from both +workflows' path filters. ### Manual Building From e05af8e81cce7a8e57b3b9075e7c49a82cf0d283 Mon Sep 17 00:00:00 2001 From: Pavel Sokolov Date: Mon, 3 Aug 2026 18:44:09 +0300 Subject: [PATCH 2/5] Apply Codex findings: fork PRs, coordinated bumps, and a skipped manifest Three findings, two of them P1, and one predates this branch. Fork pull requests were running on this org's paid runner pools. On a PR *from* a fork, github.repository_owner is the base repository - jethome-iot - so the owner check every build job carries was already true, and the fork's own Dockerfile executed on our machines. Enabling ESP-Matter on PRs would have extended that to two 180-minute builds of a fork-controlled context. Every build job now also requires github.event.pull_request.head.repo.full_name == github.repository. The PR fallback resolved a base tag that need not exist. A coordinated bump raises the ESP-IDF version and Matter's base_tag in the same branch, but the new esp-idf image is published only when that branch lands - so the fallback would have failed at FROM on precisely the change PR-5 makes. It now probes the registry and drops to :latest with a warning when the tag is not there yet, which still validates this image's own layers. On master, a failed esp-idf-build leaves esp-idf-manifest `skipped`, not `failed` - so `!= 'failure'` would have launched the Matter matrix to die downloading digest artifacts nobody created, burying the real failure under two more. master now requires the manifest to have succeeded; elsewhere it is skipped by design and only an outright failure stops the build. --- .github/workflows/esp-idf.yml | 45 +++++++++++++++++++++++++------- .github/workflows/platformio.yml | 8 ++++-- CLAUDE.md | 23 +++++++++++----- 3 files changed, 58 insertions(+), 18 deletions(-) diff --git a/.github/workflows/esp-idf.yml b/.github/workflows/esp-idf.yml index 37df23f..23622c8 100644 --- a/.github/workflows/esp-idf.yml +++ b/.github/workflows/esp-idf.yml @@ -79,7 +79,7 @@ jobs: name: 🧮 Resolve matrices runs-on: ubuntu-latest timeout-minutes: 5 - if: github.repository_owner == 'jethome-iot' + if: github.repository_owner == 'jethome-iot' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) permissions: contents: read outputs: @@ -122,7 +122,11 @@ jobs: # to jethome-iot, a fork cannot resolve their labels, and an unresolvable # `runs-on` queues for 24 hours rather than failing. A fork that wants to build # these images runs ./scripts/build.sh. - if: github.repository_owner == 'jethome-iot' + # Two conditions, not one. The owner check keeps this out of forks that run the + # workflow themselves. The head-repo check keeps it out of pull requests *from* + # a fork, where github.repository_owner is still jethome-iot - so without it a + # fork-controlled Dockerfile would execute on this org's paid runner pools. + if: github.repository_owner == 'jethome-iot' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) permissions: contents: read packages: write @@ -283,9 +287,22 @@ jobs: # esp-idf-manifest only runs on master, and `needs` on a skipped job would skip # this one too - which is why ESP-Matter went unvalidated on every pull request # while a green "🐳 ESP-IDF Docker Image" check implied otherwise. `!cancelled()` - # lets it proceed when that job was skipped, and the explicit failure test keeps - # it from building on a base whose publication actually failed. - if: ${{ !cancelled() && github.repository_owner == 'jethome-iot' && needs.esp-idf-manifest.result != 'failure' }} + # lets it proceed when that job was skipped. + # + # The two branches differ on purpose. On master the manifest must have + # *succeeded*: a failed esp-idf-build leaves the manifest job `skipped`, not + # `failed`, so accepting anything but success would start two 180-minute builds + # that then die looking for digest artifacts nobody created - burying the real + # failure. Elsewhere the manifest is skipped by design, and only an outright + # failure should stop this. + if: >- + ${{ !cancelled() + && github.repository_owner == 'jethome-iot' + && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) + && (github.ref_name == 'master' + && needs.esp-idf-manifest.result == 'success' + || github.ref_name != 'master' + && needs.esp-idf-manifest.result != 'failure') }} permissions: contents: read packages: write @@ -338,15 +355,23 @@ jobs: *) echo "::error::base digest not usable: '${digest}'"; exit 1 ;; esac ref="${IMAGE}@${digest}" - else + elif docker manifest inspect "${IMAGE}:${BASE_TAG}" >/dev/null 2>&1; then # No artifact: this is a pull request, where esp-idf-manifest does not # run and nothing has been published. Validating against the last # published base is the point - it checks this image's own layers, - # which is what the change under review touches. A PR that also edits - # esp-idf is checked against that image's *previous* release; the - # combination is only exercised once both land on master. + # which is what the change under review touches. ref="${IMAGE}:${BASE_TAG}" - echo "::notice::no digest artifact (not a master run) - validating against the published ${BASE_TAG}" + echo "::notice::not a master run - validating against the published ${BASE_TAG}" + else + # The base tag does not exist yet, which is exactly what a coordinated + # bump looks like on a pull request: this branch raises the ESP-IDF + # version and Matter's base_tag together, but the new esp-idf image is + # only published once it lands on master. Falling back to :latest still + # validates this image's own layers - the point of the PR check - where + # insisting on the unpublished tag would fail every version bump at + # `FROM`. + ref="${IMAGE}:latest" + echo "::warning::${BASE_TAG} is not published yet - validating against :latest instead. The real pairing is exercised when this lands on master." fi echo "ref=${ref}" >> "$GITHUB_OUTPUT" echo "base image: ${ref}" diff --git a/.github/workflows/platformio.yml b/.github/workflows/platformio.yml index 13eae19..e64097a 100644 --- a/.github/workflows/platformio.yml +++ b/.github/workflows/platformio.yml @@ -66,7 +66,7 @@ jobs: name: 🧮 Resolve matrices runs-on: ubuntu-latest timeout-minutes: 5 - if: github.repository_owner == 'jethome-iot' + if: github.repository_owner == 'jethome-iot' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) permissions: contents: read outputs: @@ -102,7 +102,11 @@ jobs: # to jethome-iot, a fork cannot resolve their labels, and an unresolvable # `runs-on` queues for 24 hours rather than failing. A fork that wants to build # this image runs ./scripts/build.sh. - if: github.repository_owner == 'jethome-iot' + # Two conditions, not one. The owner check keeps this out of forks that run the + # workflow themselves. The head-repo check keeps it out of pull requests *from* + # a fork, where github.repository_owner is still jethome-iot - so without it a + # fork-controlled Dockerfile would execute on this org's paid runner pools. + if: github.repository_owner == 'jethome-iot' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) permissions: contents: read packages: write diff --git a/CLAUDE.md b/CLAUDE.md index 8324b3d..e19975e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -130,12 +130,23 @@ The authoritative files are `.github/workflows/esp-idf.yml` and `idf-` tag. That validates this image's own layers, which is what a change to it touches; the *combination* of an edited esp-idf and an edited esp-matter is only exercised once both land on master. -- That fallback is why the job's `if` is written as - `${{ !cancelled() && … && needs.esp-idf-manifest.result != 'failure' }}` rather - than a plain `needs:`. A `needs` on a skipped job skips this one as well, which - is exactly how ESP-Matter went unvalidated on every pull request while a green - "🐳 ESP-IDF Docker Image" check implied otherwise. Note the `${{ }}` wrapper — - YAML reserves `!` at the start of a scalar. +- If that tag is not published either — a coordinated bump raises the ESP-IDF + version and Matter's `base_tag` together, and the new esp-idf image only exists + once it lands — the build falls back to `:latest` with a warning. Insisting on + the unpublished tag would fail every version-bump PR at `FROM`. +- That is why the job's `if` is an expression rather than a plain `needs:`. A + `needs` on a skipped job skips this one as well, which is exactly how ESP-Matter + went unvalidated while a green "🐳 ESP-IDF Docker Image" check implied otherwise. + On `master` it requires the manifest to have **succeeded**, not merely + "not failed": a failed `esp-idf-build` leaves the manifest job `skipped`, and + accepting that would start two 180-minute builds that die looking for digest + artifacts nobody created, burying the original failure. Note the `${{ }}` + wrapper — YAML reserves `!` at the start of a scalar. +- **Every build job also checks the PR's head repository**, not just the owner. On + a pull request *from* a fork `github.repository_owner` is still `jethome-iot`, so + the owner check alone would let a fork-controlled Dockerfile run on this org's + paid pools. The condition is + `github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository`. - Both workflows exclude `!images/**/*.md` from their `paths:`. Negations come last and are order-sensitive, and `paths:` cannot be mixed with `paths-ignore:` for one event. Without this a documentation-only commit rebuilds and republishes every From a3c21d09eac997c8718f195e9f6a82130182d46d Mon Sep 17 00:00:00 2001 From: Pavel Sokolov Date: Mon, 3 Aug 2026 19:21:42 +0300 Subject: [PATCH 3/5] Build ESP-Matter on the ESP-IDF this run produced, on every branch The previous commit validated ESP-Matter on pull requests but resolved its base from the registry, and the code review showed that hollowed out the very case the validation exists for. A coordinated bump raises the ESP-IDF version and Matter's base_tag in one branch; on the PR the new base is published under no tag, so the build fell back to :latest - the *old* image - and reported green. The real pairing would first meet on master, after the esp-idf tags had already been republished. esp-idf-build now pushes by digest on every run, not only on master, and esp-matter-build takes the digest for its own platform straight from that job's artifact. Nothing published this way is reachable by name: only the manifest jobs create tags and they remain master-only, so a pull request moves no tag and leaves untagged blobs behind, which want an occasional cleanup. In exchange a PR checks the pair it will actually become. This simplifies more than it adds. esp-matter-build depends on esp-idf-build rather than the master-only manifest job - hanging off that job was the original reason ESP-Matter went unvalidated on PRs - so both branches behave identically and the status-function gymnastics disappear along with the registry probe, the :latest fallback, and the manifest-digest artifact nobody reads any more. Also from the review: `!cancelled()` drops the implicit success() over *every* `needs`, so a failed prepare no longer stopped this job - it surfaced as `Unexpected end of JSON input` from an empty matrix instead of the message check-versions.sh had already written. Moot now that the condition is a plain one, but it was real. And the GHCR login in esp-matter-build is unconditional: the job pulls its base on every run, where an anonymous pull carries rate limits an authenticated one does not. --- .github/workflows/esp-idf.yml | 136 ++++++++++++---------------------- CLAUDE.md | 43 ++++------- README.md | 11 ++- 3 files changed, 70 insertions(+), 120 deletions(-) diff --git a/.github/workflows/esp-idf.yml b/.github/workflows/esp-idf.yml index 23622c8..77a936d 100644 --- a/.github/workflows/esp-idf.yml +++ b/.github/workflows/esp-idf.yml @@ -10,13 +10,13 @@ # - Multi-platform: Platform matrix builds linux/amd64 and linux/arm64 in parallel, # each on a runner of its own architecture (matrix.runner) - no QEMU # -# ESP-Matter base: on master it is the exact esp-idf digest esp-idf-manifest just -# published, handed over as an artifact within this run. On a pull request that job -# does not run, so esp-matter builds on the last published idf- tag -# instead - which validates its own layers, and is why a green check here does now -# mean ESP-Matter compiles. A PR editing both images checks each against the -# other's released version; the new combination is exercised on master. -# esp-matter-manifest stays master-only: publishing is. +# ESP-Matter builds on the esp-idf image this same run produced, on every branch: +# esp-idf-build pushes by digest always, and esp-matter-build takes the digest for +# its own platform from that job's artifact. So a green check here does mean +# ESP-Matter compiles - including on a branch that bumps both versions at once, +# where the new base exists under no tag yet. +# Only the manifest jobs write tags, and they stay master-only. A pull request +# leaves untagged blobs in GHCR; they need an occasional cleanup. name: 🐳 ESP-IDF Docker Image on: @@ -153,27 +153,33 @@ jobs: # name that the manifest job below has to look up again later, and between # those two moments another run can overwrite it. The digest is the only # thing handed forward. + # + # This pushes on pull requests too, not just master. Nothing published this + # way is reachable by name - only the manifest job creates tags, and it stays + # master-only - so a PR leaves untagged blobs behind and no moving tag. What + # it buys is that esp-matter-build can build on the ESP-IDF this very run + # produced, including on a branch that bumps both versions at once, where the + # new base does not exist under any tag yet. - name: 🐳 Build and push by digest id: build uses: docker/build-push-action@v7 with: context: images/esp-idf platforms: ${{ matrix.platform }} - outputs: type=image,name=${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.ESP_IDF_IMAGE_NAME }},push-by-digest=true,name-canonical=true,push=${{ github.ref_name == 'master' }} + outputs: type=image,name=${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.ESP_IDF_IMAGE_NAME }},push-by-digest=true,name-canonical=true,push=true build-args: ${{ matrix.build_args }} # Matrix legs cannot each set a job output - they would overwrite one # another - so the digests travel as one empty file per leg, named after the - # digest itself. + # digest itself. Uploaded on every run: esp-matter-build consumes this on + # pull requests as well. - name: 📤 Export digest - if: github.ref_name == 'master' run: | mkdir -p /tmp/digests digest="${{ steps.build.outputs.digest }}" touch "/tmp/digests/${digest#sha256:}" - name: ⬆️ Upload digest - if: github.ref_name == 'master' uses: actions/upload-artifact@v7 with: name: digest-${{ env.ESP_IDF_IMAGE_NAME }}-${{ matrix.tag }}-${{ matrix.platform_tag }} @@ -252,57 +258,31 @@ jobs: tags+=(-t "${IMAGE}:${TAG}") docker buildx imagetools create "${tags[@]}" "${refs[@]}" - # Hand the manifest's own digest to esp-matter-build, so it builds FROM - # the exact image this run published rather than whatever the tag points - # at by the time it starts. It travels as an artifact rather than a job - # output because this job has a matrix: a scalar output would be - # overwritten by whichever leg finished last, silently pinning - # esp-matter to an arbitrary ESP-IDF version once there is more than one. + # Reported, not handed on: esp-matter-build takes the per-platform + # digests from esp-idf-build directly, so nothing downstream waits on + # this job any more. digest=$(docker buildx imagetools inspect "${IMAGE}:${TAG}" \ --format '{{json .Manifest}}' | jq -er '.digest') - case "${digest}" in - sha256:*) ;; - *) echo "::error::manifest digest not resolved: '${digest}'"; exit 1 ;; - esac - mkdir -p /tmp/manifest - printf '%s' "${digest}" > /tmp/manifest/digest echo "published ${IMAGE}:${TAG} as ${digest}" - - name: ⬆️ Upload base image digest - uses: actions/upload-artifact@v7 - with: - name: manifest-digest-${{ env.ESP_IDF_IMAGE_NAME }}-${{ matrix.tag }} - path: /tmp/manifest/digest - if-no-files-found: error - retention-days: 1 - esp-matter-build: name: esp-matter-build (${{ matrix.tag }}, ${{ matrix.platform }}) - needs: [prepare, esp-idf-manifest] + needs: [prepare, esp-idf-build] runs-on: ${{ matrix.runner }} # From images/versions.json. A cap on a wedged job, not a target: killing this # leg skips the manifest job with it and leaves the published tags on the # previous build while the other platform's image sits in GHCR unreferenced. timeout-minutes: ${{ matrix.timeout_minutes }} - # esp-idf-manifest only runs on master, and `needs` on a skipped job would skip - # this one too - which is why ESP-Matter went unvalidated on every pull request - # while a green "🐳 ESP-IDF Docker Image" check implied otherwise. `!cancelled()` - # lets it proceed when that job was skipped. - # - # The two branches differ on purpose. On master the manifest must have - # *succeeded*: a failed esp-idf-build leaves the manifest job `skipped`, not - # `failed`, so accepting anything but success would start two 180-minute builds - # that then die looking for digest artifacts nobody created - burying the real - # failure. Elsewhere the manifest is skipped by design, and only an outright - # failure should stop this. + # Depends on esp-idf-build, not esp-idf-manifest. The manifest job publishes + # tags and is master-only, so hanging off it is what kept ESP-Matter + # unvalidated on every pull request while a green "🐳 ESP-IDF Docker Image" + # check implied otherwise. What this job actually needs is the ESP-IDF image + # for its own platform, which the build leg pushes by digest on every run - so + # the dependency is plain, both branches behave identically, and no status + # expression is needed to work around a skipped job. if: >- - ${{ !cancelled() - && github.repository_owner == 'jethome-iot' - && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) - && (github.ref_name == 'master' - && needs.esp-idf-manifest.result == 'success' - || github.ref_name != 'master' - && needs.esp-idf-manifest.result != 'failure') }} + ${{ github.repository_owner == 'jethome-iot' + && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) }} permissions: contents: read packages: write @@ -321,60 +301,42 @@ jobs: - name: 🔧 Set up Docker Buildx uses: docker/setup-buildx-action@v4 + # Unconditional, unlike the other build jobs: this one pulls its base from + # GHCR on every run, including pull requests, and an anonymous pull is + # subject to rate limits the authenticated one is not. - name: 🔐 Log in to GitHub Container Registry - if: github.ref_name == 'master' uses: docker/login-action@v4 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - # On master the base is the digest this very run published, downloaded from an - # artifact named after the ESP-IDF version this image declares in its own tag - # - so the base it builds on and the version it advertises cannot drift - # apart, and a half-done bump fails here instead of publishing an idf-v - # tag built on v. - - name: ⬇️ Resolve base image digest - if: github.ref_name == 'master' + # The ESP-IDF image for this platform, pushed by digest by esp-idf-build in + # this same run. Not the multi-arch manifest: that is published only on + # master, and this leg builds one platform anyway. Naming the artifact after + # the ESP-IDF version this image declares in its own tag keeps the base it + # stands on and the version it advertises from drifting apart - a half-done + # bump fails here rather than publishing an idf-v tag built on v. + - name: ⬇️ Download base image digest uses: actions/download-artifact@v8 with: - name: manifest-digest-${{ env.ESP_IDF_IMAGE_NAME }}-${{ matrix.base_tag }} + name: digest-${{ env.ESP_IDF_IMAGE_NAME }}-${{ matrix.base_tag }}-${{ matrix.platform_tag }} path: /tmp/base - name: 🏷️ Resolve base image reference id: base env: IMAGE: ${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.ESP_IDF_IMAGE_NAME }} - BASE_TAG: ${{ matrix.base_tag }} run: | set -euo pipefail - if [ -f /tmp/base/digest ]; then - digest=$(cat /tmp/base/digest) - case "${digest}" in - sha256:*) ;; - *) echo "::error::base digest not usable: '${digest}'"; exit 1 ;; - esac - ref="${IMAGE}@${digest}" - elif docker manifest inspect "${IMAGE}:${BASE_TAG}" >/dev/null 2>&1; then - # No artifact: this is a pull request, where esp-idf-manifest does not - # run and nothing has been published. Validating against the last - # published base is the point - it checks this image's own layers, - # which is what the change under review touches. - ref="${IMAGE}:${BASE_TAG}" - echo "::notice::not a master run - validating against the published ${BASE_TAG}" - else - # The base tag does not exist yet, which is exactly what a coordinated - # bump looks like on a pull request: this branch raises the ESP-IDF - # version and Matter's base_tag together, but the new esp-idf image is - # only published once it lands on master. Falling back to :latest still - # validates this image's own layers - the point of the PR check - where - # insisting on the unpublished tag would fail every version bump at - # `FROM`. - ref="${IMAGE}:latest" - echo "::warning::${BASE_TAG} is not published yet - validating against :latest instead. The real pairing is exercised when this lands on master." - fi - echo "ref=${ref}" >> "$GITHUB_OUTPUT" - echo "base image: ${ref}" + # One file, named after the digest it stands for. + digest=$(find /tmp/base -type f -printf '%f\n' | head -1) + case "${digest}" in + [0-9a-f]*) ;; + *) echo "::error::no usable base digest in the artifact"; exit 1 ;; + esac + echo "ref=${IMAGE}@sha256:${digest}" >> "$GITHUB_OUTPUT" + echo "base image: ${IMAGE}@sha256:${digest}" - name: 🐳 Build and push by digest id: build diff --git a/CLAUDE.md b/CLAUDE.md index e19975e..1fffc3f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -59,14 +59,13 @@ The authoritative files are `.github/workflows/esp-idf.yml` and There are no `-linux-` tags in GHCR any more. - The version tag is listed **last** in `imagetools create` on purpose: GHCR shows the last tag as the package's primary tag. -- `esp-idf-manifest` publishes the manifest's own digest as an artifact named - `manifest-digest--`, and `esp-matter-build` downloads the one - matching the ESP-IDF version in its own tag, then builds `FROM` it via a single - `BASE_IMAGE` build-arg. An artifact rather than a job output because that job has - a matrix: a scalar output would be overwritten by whichever leg finished last. - One argument rather than repo + tag because a digest needs `@`, not `:` — and it - makes the base repository overridable, so a fork or a local build can point at - its own esp-idf instead of this one. +- `esp-matter-build` depends on `esp-idf-build`, **not** on the manifest job, and + takes the per-platform digest for its own architecture from that build's + artifact. The artifact is named after the ESP-IDF version this image declares in + its own tag, so base and advertised version cannot drift apart. It reaches the + Dockerfile through a single `BASE_IMAGE` build-arg — one argument rather than + repo + tag, because a digest needs `@` where a tag needs `:`, and because it makes + the base repository overridable for a fork or a local build. - **`images/versions.json` is the single source of truth for what CI builds.** A `prepare` job runs `scripts/check-versions.sh`, then `scripts/versions-matrix.sh build|manifest`, and every other job takes its matrix from @@ -123,25 +122,15 @@ The authoritative files are `.github/workflows/esp-idf.yml` and built and thrown away, never uploaded. Manifest jobs ask both in their own job `if`, since they have no build step to gate. The org literal is hardcoded. -- **`esp-matter-build` runs on pull requests too**, and the base it uses differs by - context. On `master` it downloads the digest artifact `esp-idf-manifest` just - published and builds on that exact image. Elsewhere no such artifact exists — the - manifest job is master-only — so it falls back to the last published - `idf-` tag. That validates this image's own layers, which is what a - change to it touches; the *combination* of an edited esp-idf and an edited - esp-matter is only exercised once both land on master. -- If that tag is not published either — a coordinated bump raises the ESP-IDF - version and Matter's `base_tag` together, and the new esp-idf image only exists - once it lands — the build falls back to `:latest` with a warning. Insisting on - the unpublished tag would fail every version-bump PR at `FROM`. -- That is why the job's `if` is an expression rather than a plain `needs:`. A - `needs` on a skipped job skips this one as well, which is exactly how ESP-Matter - went unvalidated while a green "🐳 ESP-IDF Docker Image" check implied otherwise. - On `master` it requires the manifest to have **succeeded**, not merely - "not failed": a failed `esp-idf-build` leaves the manifest job `skipped`, and - accepting that would start two 180-minute builds that die looking for digest - artifacts nobody created, burying the original failure. Note the `${{ }}` - wrapper — YAML reserves `!` at the start of a scalar. +- **Every image is build-validated on pull requests**, ESP-Matter included, and on + the ESP-IDF this very run produced — so a branch bumping both versions at once is + checked as the pair it will become, not against whatever is published today. +- That works because `esp-idf-build` pushes by digest on **every** run, not only on + master. Nothing published that way is reachable by name: only the manifest jobs + create tags, and they stay master-only. A pull request therefore leaves untagged + blobs in GHCR and moves no tag. They accumulate; clean them up periodically. +- Publishing is still master-only in the sense that matters — no tag, `latest` + included, is ever written outside master. - **Every build job also checks the PR's head repository**, not just the owner. On a pull request *from* a fork `github.repository_owner` is still `jethome-iot`, so the owner check alone would let a fork-controlled Dockerfile run on this org's diff --git a/README.md b/README.md index f79ca5c..a97ac93 100644 --- a/README.md +++ b/README.md @@ -135,12 +135,11 @@ instead of the versions CI passes in, so a green local build is not a green CI run. **Note:** every image is build-validated on pull requests and on `dev`, including -one built `FROM` another image of this repo. What differs is the base it is built -on. On `master`, ESP-Matter builds on the exact ESP-IDF digest the same run just -published; anywhere else that digest does not exist yet, so it builds on the last -published `idf-` tag. A pull request therefore proves that the image -compiles, but a PR editing *both* images only proves each against the other's -released version — the new combination is exercised when they land on `master`. +one built `FROM` another image of this repo, and on the base that same run +produced — so a change touching both images is checked as the pair it will become. +This works because build jobs push by digest on every run; those pushes carry no +tag, and only `master` ever writes `latest` or a version tag. The untagged blobs a +pull request leaves in GHCR accumulate and want an occasional cleanup. Documentation-only changes trigger nothing: `!images/**/*.md` is excluded from both workflows' path filters. From 9dce8cfb55c999534c9f1f8f4a1071892d2d6c46 Mon Sep 17 00:00:00 2001 From: Pavel Sokolov Date: Mon, 3 Aug 2026 19:27:07 +0300 Subject: [PATCH 4/5] Log in on every run, and let Dependabot past the digest push Two findings from Codex, both about the push that the previous commit made unconditional. The login in esp-idf-build was still gated on master while push=true had become unconditional, so every dev push, same-repo PR and non-master dispatch would have failed at the push with no credentials - taking the ESP-Matter validation this series just added down with it, since no digest artifact would exist. The login is unconditional now, matching the push. Dependabot cannot push packages at all: GitHub hands its runs a read-only GITHUB_TOKEN whatever the permissions block asks for. Since Dependabot was enabled in this same series and its PRs edit workflow files - which are in the paths filter - this would have failed on the first automated bump. Those runs now skip the push and the digest upload, and esp-matter-build sits them out. The action bump is still exercised: esp-idf-build compiles the image, which is what such a PR changes. --- .github/workflows/esp-idf.yml | 18 ++++++++++++++++-- CLAUDE.md | 12 +++++++++--- 2 files changed, 25 insertions(+), 5 deletions(-) diff --git a/.github/workflows/esp-idf.yml b/.github/workflows/esp-idf.yml index 77a936d..0622d20 100644 --- a/.github/workflows/esp-idf.yml +++ b/.github/workflows/esp-idf.yml @@ -141,8 +141,9 @@ jobs: - name: 🔧 Set up Docker Buildx uses: docker/setup-buildx-action@v4 + # Unconditional: the build below pushes by digest on every run, so every run + # needs credentials - not just master. - name: 🔐 Log in to GitHub Container Registry - if: github.ref_name == 'master' uses: docker/login-action@v4 with: registry: ${{ env.REGISTRY }} @@ -160,13 +161,19 @@ jobs: # it buys is that esp-matter-build can build on the ESP-IDF this very run # produced, including on a branch that bumps both versions at once, where the # new base does not exist under any tag yet. + # + # Except for Dependabot: GitHub hands its runs a read-only GITHUB_TOKEN + # whatever the permissions block asks for, so the push would be rejected. Its + # pull requests bump action versions, which this job still exercises by + # building; only the upload is skipped, and esp-matter-build sits that run out + # (see its `if`). - name: 🐳 Build and push by digest id: build uses: docker/build-push-action@v7 with: context: images/esp-idf platforms: ${{ matrix.platform }} - outputs: type=image,name=${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.ESP_IDF_IMAGE_NAME }},push-by-digest=true,name-canonical=true,push=true + outputs: type=image,name=${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.ESP_IDF_IMAGE_NAME }},push-by-digest=true,name-canonical=true,push=${{ github.actor != 'dependabot[bot]' }} build-args: ${{ matrix.build_args }} # Matrix legs cannot each set a job output - they would overwrite one @@ -174,12 +181,14 @@ jobs: # digest itself. Uploaded on every run: esp-matter-build consumes this on # pull requests as well. - name: 📤 Export digest + if: github.actor != 'dependabot[bot]' run: | mkdir -p /tmp/digests digest="${{ steps.build.outputs.digest }}" touch "/tmp/digests/${digest#sha256:}" - name: ⬆️ Upload digest + if: github.actor != 'dependabot[bot]' uses: actions/upload-artifact@v7 with: name: digest-${{ env.ESP_IDF_IMAGE_NAME }}-${{ matrix.tag }}-${{ matrix.platform_tag }} @@ -280,8 +289,13 @@ jobs: # for its own platform, which the build leg pushes by digest on every run - so # the dependency is plain, both branches behave identically, and no status # expression is needed to work around a skipped job. + # + # Dependabot is excluded: its token cannot push, so esp-idf-build produces no + # digest to hand over. Its pull requests change action versions, which + # esp-idf-build still exercises. if: >- ${{ github.repository_owner == 'jethome-iot' + && github.actor != 'dependabot[bot]' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) }} permissions: contents: read diff --git a/CLAUDE.md b/CLAUDE.md index 1fffc3f..9c48039 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -126,9 +126,15 @@ The authoritative files are `.github/workflows/esp-idf.yml` and the ESP-IDF this very run produced — so a branch bumping both versions at once is checked as the pair it will become, not against whatever is published today. - That works because `esp-idf-build` pushes by digest on **every** run, not only on - master. Nothing published that way is reachable by name: only the manifest jobs - create tags, and they stay master-only. A pull request therefore leaves untagged - blobs in GHCR and moves no tag. They accumulate; clean them up periodically. + master — so its GHCR login is unconditional too, unlike the other build jobs'. + Nothing published that way is reachable by name: only the manifest jobs create + tags, and they stay master-only. A pull request therefore leaves untagged blobs in + GHCR and moves no tag. They accumulate; clean them up periodically. +- **Dependabot is the exception.** GitHub gives its runs a read-only + `GITHUB_TOKEN` regardless of the job's `permissions:`, so the digest push would + be rejected. Its pull requests skip the push and the digest upload, and + `esp-matter-build` sits them out — the action bump they carry is still exercised + by esp-idf-build compiling the image. - Publishing is still master-only in the sense that matters — no tag, `latest` included, is ever written outside master. - **Every build job also checks the PR's head repository**, not just the owner. On From bab135c3ea8793173f7a39a562b8c155f63f9b74 Mon Sep 17 00:00:00 2001 From: Pavel Sokolov Date: Mon, 3 Aug 2026 19:46:16 +0300 Subject: [PATCH 5/5] Say which job pushes on every run, and which does not Copilot caught two places left describing the old behaviour. The esp-idf.yml header still said dev/PR runs were build-only, which stopped being true when esp-idf-build started pushing by digest so esp-matter-build could consume the artifact. And the README generalised that to all build jobs, when it is specific to ESP-IDF: it is the only image another one is built from, so the only one with a reason to push outside master. PlatformIO still uploads nothing there, and its header now says why. --- .github/workflows/esp-idf.yml | 11 ++++++++--- .github/workflows/platformio.yml | 3 ++- README.md | 9 ++++++--- 3 files changed, 16 insertions(+), 7 deletions(-) diff --git a/.github/workflows/esp-idf.yml b/.github/workflows/esp-idf.yml index 0622d20..222daf3 100644 --- a/.github/workflows/esp-idf.yml +++ b/.github/workflows/esp-idf.yml @@ -2,9 +2,14 @@ # Builds ESP-IDF development image with pytest, QEMU, and testing tools # # Build Strategy: -# - Main repo master: Build + Push to GHCR -# - Main repo dev/PR: Build only (validation) -# - workflow_dispatch: Build on any branch of the main repo, push only on master +# - Tags (latest, sha-, -) are written on master only, by +# the manifest jobs. +# - Images themselves are pushed by digest on EVERY run, master or not, carrying no +# tag and reachable by nothing but that digest. esp-matter-build consumes them, +# which is what lets it validate on a pull request. The cost is untagged blobs +# accumulating in GHCR. +# - Dependabot is the exception: its GITHUB_TOKEN is read-only, so those runs build +# without pushing and esp-matter-build sits them out. # - Forks: nothing runs - the runner pools are not reachable there; use # ./scripts/build.sh instead # - Multi-platform: Platform matrix builds linux/amd64 and linux/arm64 in parallel, diff --git a/.github/workflows/platformio.yml b/.github/workflows/platformio.yml index e64097a..2714187 100644 --- a/.github/workflows/platformio.yml +++ b/.github/workflows/platformio.yml @@ -2,7 +2,8 @@ # # Build Strategy: # - Main repo master: Build + Push to GHCR -# - Main repo dev/PR: Build only (validation) +# - Main repo dev/PR: Build only (validation) - nothing is uploaded. Unlike +# esp-idf.yml, no other job consumes this image, so there is nothing to push for. # - workflow_dispatch: Build on any branch of the main repo, push only on master # - Forks: nothing runs - the runner pools are not reachable there; use # ./scripts/build.sh instead diff --git a/README.md b/README.md index a97ac93..4fb30c9 100644 --- a/README.md +++ b/README.md @@ -137,9 +137,12 @@ run. **Note:** every image is build-validated on pull requests and on `dev`, including one built `FROM` another image of this repo, and on the base that same run produced — so a change touching both images is checked as the pair it will become. -This works because build jobs push by digest on every run; those pushes carry no -tag, and only `master` ever writes `latest` or a version tag. The untagged blobs a -pull request leaves in GHCR accumulate and want an occasional cleanup. +This works because the **ESP-IDF** build job pushes by digest on every run — it is +the only image another one is built from, so it is the only one that needs to. +Those pushes carry no tag, and only `master` ever writes `latest` or a version tag. +The untagged blobs a pull request leaves in GHCR accumulate and want an occasional +cleanup. PlatformIO, which nothing builds on, still uploads nothing outside +`master`. Documentation-only changes trigger nothing: `!images/**/*.md` is excluded from both workflows' path filters.