Skip to content

capi: add qemu node conformance validation hook - #2101

Open
MaxRink wants to merge 10 commits into
kubernetes-sigs:mainfrom
MaxRink:capi/node-conformance-qemu-validation
Open

capi: add qemu node conformance validation hook#2101
MaxRink wants to merge 10 commits into
kubernetes-sigs:mainfrom
MaxRink:capi/node-conformance-qemu-validation

Conversation

@MaxRink

@MaxRink MaxRink commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

  • Adds an opt-in QEMU Packer hook for Kubernetes e2e-node conformance validation.
  • Downloads the version-matched Kubernetes test tarball and runs e2e_node.test with a [Conformance] focus.
  • Downloads node-conformance-results/ before failing the build when conformance fails.
  • Preserves pre-existing CRI images during node-conformance cleanup so image preload checks remain meaningful.
  • Adds images/capi/scripts/ci-qemu-node-conformance.sh as a nested-virtualization CI helper with explicit CPU, memory, timeout, parallelism, accelerator, and target overrides.
  • Documents the config flags and QEMU usage.
  • Keeps the hook disabled by default and documents it as release or periodic image validation, not routine presubmit validation.

Related CI plumbing: kubernetes/test-infra#37401

Validation

  • bash -n images/capi/hack/run-e2e-node-conformance.sh images/capi/scripts/ci-qemu-node-conformance.sh
  • shellcheck images/capi/hack/run-e2e-node-conformance.sh images/capi/scripts/ci-qemu-node-conformance.sh
  • jq empty images/capi/packer/config/node-conformance.json images/capi/packer/qemu/packer.json
  • make validate-qemu-ubuntu-2404-cloudimg PACKER_FLAGS="--var 'node_conformance=true' --var 'accelerator=tcg' --var 'cpu_model=max' --var 'cpus=4' --var 'memory=4096'"
  • NODE_CONFORMANCE_TARGET=validate-qemu-ubuntu-2404-cloudimg NODE_CONFORMANCE_ACCELERATOR=tcg NODE_CONFORMANCE_CPU_MODEL=max images/capi/scripts/ci-qemu-node-conformance.sh
  • git diff --check
  • For the test-infra periodic job draft: YAML parse for config/jobs/kubernetes-sigs/image-builder/*.yaml and go test ./config/tests/jobs/...

@kubernetes-prow

kubernetes-prow Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@kubernetes-prow kubernetes-prow Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 6, 2026
@kubernetes-prow

kubernetes-prow Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jsturtevant for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 6, 2026
@MaxRink

MaxRink commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

/test all

@MaxRink
MaxRink marked this pull request as ready for review July 6, 2026 22:58
@kubernetes-prow kubernetes-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 6, 2026
@kubernetes-prow
kubernetes-prow Bot requested review from AndiDog and davidjumani July 6, 2026 22:58
@MaxRink
MaxRink force-pushed the capi/node-conformance-qemu-validation branch from d85a5b8 to ca164db Compare July 6, 2026 23:24
@kubernetes-prow kubernetes-prow Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 7, 2026
@MaxRink MaxRink mentioned this pull request Jul 7, 2026
Draft
@MaxRink
MaxRink force-pushed the capi/node-conformance-qemu-validation branch 4 times, most recently from ee0c490 to de282cc Compare July 8, 2026 00:08
drew-viles
drew-viles reviewed Jul 16, 2026
View reviewed changes

@drew-viles drew-viles left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor notes

Comment thread images/capi/packer/config/node-conformance.json Outdated
Comment thread images/capi/packer/qemu/packer.json.tmpl Outdated
@MaxRink
MaxRink force-pushed the capi/node-conformance-qemu-validation branch from de282cc to 8b19783 Compare July 17, 2026 16:15
@drew-viles

drew-viles commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

/retest

@mboersma

mboersma commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

/label tide/merge-method-squash

@kubernetes-prow kubernetes-prow Bot added tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jul 19, 2026
@MaxRink
MaxRink force-pushed the capi/node-conformance-qemu-validation branch from 8b19783 to 4645be4 Compare July 28, 2026 16:06
@kubernetes-prow kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 28, 2026
@MaxRink
MaxRink force-pushed the capi/node-conformance-qemu-validation branch from 4645be4 to 18127bd Compare August 9, 2026 14:45
@vasartori

vasartori commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

/test pull-azure-sigs

@vasartori

vasartori commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Sorry about the retest command...
I interacted with the wrong PR 😕

mboersma
mboersma requested changes Aug 10, 2026
View reviewed changes

@mboersma mboersma left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we avoid leaving conformance-test state in the final image? The hook runs after sysprep/Goss and currently leaves test files and potentially kubelet/CNI/runtime state behind. Please run it against a disposable copy or fully clean and restore the image before shutdown.

Also, please update the documented defaults and either support or explicitly exclude Flatcar.

@linux-foundation-easycla

linux-foundation-easycla Bot commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

@kubernetes-prow kubernetes-prow Bot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 11, 2026
@MaxRink
MaxRink force-pushed the capi/node-conformance-qemu-validation branch from 6fdfb0a to c43b35c Compare August 12, 2026 00:59
@kubernetes-prow kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Aug 12, 2026
MaxRink added 9 commits August 22, 2026 16:06
@MaxRink
capi: add qemu node conformance validation hook
b1c9e1e
@MaxRink
capi: preserve CRI images during node conformance
eb2b474
@MaxRink
capi: add node conformance CI helper
6583566
@MaxRink
capi: always overwrite node-conformance summary.env on failure
e0b3c3b 
A stale summary.env with exit_code=0 from a previously preserved or
custom results directory could mask a real conformance failure: the
provisioner only wrote summary.env when the file did not already
exist, so a pre-existing summary always won over the current run's
result. Always overwrite summary.env with the current exit code so a
stale prior result can never hide a failing run.
@MaxRink
capi: run node conformance Serial tests by default
520de0b 
The default skip expression excluded [Serial] tests, so genuine
node-conformance cases tagged serial (e.g. MirrorPod coverage) never
ran, yet the hook could still report success as a complete node
conformance result. Default Ginkgo to a single node and drop [Serial]
from the default skip expression so serial cases run instead of being
silently skipped.
@MaxRink
capi: clean node conformance image state
b64e138 
Snapshot/restore kubelet and CNI paths around the node-conformance run, isolate temporary kubelet/CNI data, and clean runner artifacts so final images do not retain conformance state. Add explicit Flatcar exclusion and sync docs/defaults, plus targeted unit tests for the hook and template cleanup behavior.
@MaxRink
capi: harden node conformance cleanup
af0c3a6
@MaxRink
capi: make node conformance hook shell portable
35ee6d2
@MaxRink
ci: retry pull-ova-all
5656049
@MaxRink
MaxRink force-pushed the capi/node-conformance-qemu-validation branch from 6cd3649 to 5656049 Compare August 22, 2026 14:08
@kubernetes-prow kubernetes-prow Bot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 22, 2026
@MaxRink

MaxRink commented Aug 22, 2026

Copy link
Copy Markdown
Contributor Author

Addressed mboersma's review feedback in commit 9c4764dcd0644e563a54cbf0defc5523c15e6e93:

  1. Cleanup/lifecycle: the hook now removes its disposable work directory on every exit, including failed runs; snapshots/restores kubelet/CNI/runtime filesystem state; and restores the pre-run kubelet, containerd, and CRI-O service activity. Packer downloads reports first, removes the guest-side script/results before validation, and validates the downloaded summary locally, so a failed validation cannot bypass guest cleanup.
  2. Documented defaults: the node-conformance documentation now covers the helper's KVM/host/4-CPU/8-GiB defaults and all override variables. The focused test verifies every node-conformance.json default is represented in the docs.
  3. Flatcar: node conformance remains explicitly unsupported on Flatcar. The CI helper rejects Flatcar targets before building, and the guest hook rejects Flatcar via /etc/os-release; documentation and focused tests cover both paths.

Changed files: run-e2e-node-conformance.sh, ci-qemu-node-conformance.sh, packer/qemu/packer.json.tmpl, node-conformance_hook_test.py, and the node-conformance/QEMU README documentation.

Validation passed:

  • make test-qemu-immutable (37 tests)
  • focused node-conformance tests (16 tests)
  • shellcheck and bash -n for changed scripts
  • Packer validation for qemu-ubuntu-2404
  • git diff --check

Please re-review.

@MaxRink
MaxRink force-pushed the capi/node-conformance-qemu-validation branch from 9c4764d to a0201fb Compare August 22, 2026 16:37
@kubernetes-prow kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@drew-viles drew-viles drew-viles left review comments
@mboersma mboersma mboersma requested changes
@weizhouapache weizhouapache Awaiting requested review from weizhouapache
@AndiDog AndiDog Awaiting requested review from AndiDog
@davidjumani davidjumani Awaiting requested review from davidjumani

Assignees

No one assigned

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@MaxRink @drew-viles @mboersma @vasartori