Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/book/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
- [MaaS](./capi/providers/maas.md)
- [Including ECR Credential Provider](./capi/ecr-credential-provider.md)
- [Testing the Images](./capi/goss/goss.md)
- [Kubernetes Node Conformance](./capi/node-conformance.md)
- [Using Container Images](./capi/container-image.md)
- [Customizing containerd](./capi/containerd/customizing-containerd.md)
- [Kubernetes version matrix](./capi/kubernetes-version-matrix.md)
Expand Down
94 changes: 94 additions & 0 deletions docs/book/src/capi/node-conformance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Kubernetes Node Conformance

Image Builder can run the Kubernetes `e2e_node.test` conformance subset as an
optional QEMU image validation step after the normal Goss checks.

The hook is disabled by default because it downloads the version-matched
Kubernetes test tarball and adds significant runtime. Enable it for manual
validation when you need stronger signal that a newly built node image can run a
conformant kubelet and container runtime.

## QEMU Usage

Run a QEMU build with node conformance enabled:

```bash
cd images/capi
PACKER_FLAGS="--var 'node_conformance=true'" make build-qemu-ubuntu-2404
```

From the repository root, the CI helper wraps the same configuration with
defaults suitable for a nested-virtualization runner:

```bash
images/capi/scripts/ci-qemu-node-conformance.sh
```

The helper builds `build-qemu-ubuntu-2404-cloudimg` by default with
`node_conformance=true`, KVM acceleration, the host CPU model, 4 CPUs, and
8 GiB of memory. Override
`NODE_CONFORMANCE_TARGET`, `NODE_CONFORMANCE_CPUS`,
`NODE_CONFORMANCE_MEMORY`, `NODE_CONFORMANCE_PARALLELISM`,
`NODE_CONFORMANCE_TIMEOUT`, `NODE_CONFORMANCE_ACCELERATOR`, or
`NODE_CONFORMANCE_CPU_MODEL` to tune a run. The helper requires `/dev/kvm`
unless `NODE_CONFORMANCE_ACCELERATOR=tcg` is set explicitly for slower local
debugging.

The runner downloads `kubernetes-test-linux-${ARCH}.tar.gz` for the same
Kubernetes version configured by `kubernetes_semver`, starts the local CRI
runtime, stops the system kubelet, and runs `e2e_node.test` with a default focus
of `[Conformance]`.

Results are downloaded to `node-conformance-results/` before Packer evaluates
the test result. This preserves logs and JUnit/report files even when the
conformance run fails. Before validation, Packer removes the guest-side test
script and result directory, so a failed validation cannot leave those files in
the image. The hook snapshots and restores kubelet/CNI and runtime filesystem
state, restores service activity, removes only runtime resources created by the
test, and restarts pre-existing running containers before shutdown.

Flatcar targets are currently excluded: setting `node_conformance=true` on a
Flatcar QEMU build fails with an explicit unsupported message.

## Configuration

The defaults live in `packer/config/node-conformance.json` and can be overridden
with `PACKER_FLAGS` or an additional Packer var file.

| Variable | Default | Description |
| --- | --- | --- |
| `node_conformance` | `false` | Enables the QEMU node conformance hook. |
| `node_conformance_focus` | `\[Conformance\]` | Ginkgo focus expression. |
| `node_conformance_skip` | `\[Flaky\]\|\[Slow\]` | Ginkgo skip expression. |
| `node_conformance_parallelism` | `1` | Ginkgo parallel node count. |
| `node_conformance_timeout` | `2h` | Ginkgo timeout for the e2e-node run. |
| `node_conformance_flake_attempts` | `1` | Ginkgo flake attempts. |
| `node_conformance_standalone_mode` | `true` | Passes `--standalone-mode=true` to `e2e_node.test`. |
| `node_conformance_kubelet_flags` | `--fail-swap-on=false --runtime-cgroups=/system.slice/containerd.service` | Extra kubelet flags passed to `e2e_node.test`. |
| `node_conformance_etcd_version` | `v3.5.32` | etcd version downloaded when `etcd` is not already installed. |
| `node_conformance_results_dir` | `/tmp/kubernetes-node-conformance-results` | Guest result directory downloaded by Packer. |

`node_conformance` accepts case-insensitive `true`/`false`, `yes`/`no`, and
`1`/`0` values. Packer normalizes the value before running the hook and before
evaluating its result.

Example with custom focus and fewer parallel nodes:

```bash
cd images/capi
PACKER_FLAGS="--var 'node_conformance=true' \
--var 'node_conformance_parallelism=2' \
--var 'node_conformance_focus=\\[Conformance\\]'" \
make build-qemu-ubuntu-2404
```

## Scope

Node conformance validates a node image in isolation. It complements Goss image
checks, but it does not replace Cluster API provider e2e tests or Kubernetes
cluster conformance suites that need a bootstrapped cluster.

References:

- Kubernetes node conformance: <https://kubernetes.io/docs/setup/best-practices/node-conformance/>
- SIG Node e2e-node tests: <https://github.com/kubernetes/community/blob/main/contributors/devel/sig-node/e2e-node-tests.md>
1 change: 1 addition & 0 deletions images/capi/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ manifest.json
# Goss test droppings
debug-goss-spec.yaml
goss-spec.yaml
node-conformance-results/
3 changes: 2 additions & 1 deletion images/capi/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,7 @@ COMMON_NODE_VAR_FILES := packer/config/kubernetes.json \
packer/config/wasm-shims.json \
packer/config/ansible-args.json \
packer/config/goss-args.json \
packer/config/node-conformance.json \
packer/config/common.json \
packer/config/additional_components.json \
packer/config/ecr_credential_provider.json
Expand Down Expand Up @@ -621,7 +622,7 @@ $(QEMU_VALIDATE_TARGETS): deps-qemu set-ssh-password
$(PACKER) validate $(PACKER_NODE_FLAGS) -var-file="$(abspath packer/qemu/$(subst validate-,,$@).json)" $(ABSOLUTE_PACKER_VAR_FILES) packer/qemu/packer.json

.PHONY: test-qemu-immutable
test-qemu-immutable: ## Runs immutable QEMU helper unit tests
test-qemu-immutable: ## Runs QEMU helper unit tests
python3 -m unittest discover -s packer/qemu/scripts -p '*_test.py'

validate-qemu-ubuntu-2404-immutable: test-qemu-immutable
Expand Down
Loading