[WIP] Run Kubernetes node conformance tests in Build Azure SIG Image workflow - #2013
Draft
mboersma wants to merge 7 commits into
Draft
[WIP] Run Kubernetes node conformance tests in Build Azure SIG Image workflow#2013mboersma wants to merge 7 commits into
mboersma wants to merge 7 commits into
Conversation
Adds an opt-in, non-blocking step to the Build Azure SIG Image workflow that runs the [NodeConformance] subset of e2e_node.test against the freshly-booted test VM. - New optional workflow input run_node_conformance (default false). - New helper script images/capi/hack/node-conformance.sh that downloads the version-matched kubernetes-test tarball from dl.k8s.io and runs e2e_node.test in standalone mode against the system containerd. - Linux only; Windows is skipped (node conformance is Linux-only and the Windows e2e subset requires a real cluster). - Step is continue-on-error so it surfaces signal as a warning without blocking promote while we iterate. - Results are uploaded as the node-conformance-results artifact. - Test job timeout-minutes bumped to 150 only when conformance is on.
Contributor
|
Skipping CI for Draft Pull Request. |
k8s-ci-robot
added
the
do-not-merge/work-in-progress
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
k8s-ci-robot
added
cncf-cla: yes
Fetch the matching .sha256 file from dl.k8s.io and verify the test tarball before extracting it. The published checksum file contains only the hex digest, so we synthesize a sha256sum-compatible line and use sha256sum --check --strict for verification.
The runtime cgroup path was hardcoded to containerd. When CRI-O is the detected runtime, set --runtime-cgroups to /system.slice/crio.service instead so kubelet tracks the correct slice.
Previously the node-conformance step used continue-on-error: true, which masked all failures equally — including SSH/SCP transport, auth, and network problems — as ordinary conformance test failures. Drop continue-on-error and handle exit codes explicitly: - scp helper-up failures fail the step (transport error). - ssh exit 255 (its own transport/auth errors) fails the step. - Any other non-zero remote exit is treated as a conformance result and surfaced as a workflow warning without failing the job. - scp results-back failure is a warning only (we already have a conformance verdict at that point).
systemd materializes a slice only when something is first launched into it. On a fresh image-builder VM the system kubelet has never been started, so /kubelet.slice may not exist when the test kubelet is launched. Letting kubelet pick its own cgroup avoids a needless assumption and works on systems where the slice does exist as well.
First real run surfaced three issues: 1. e2e_node.test died with "Could not locate kubelet binary" because the framework searches dirname(argv[0]) and a Kubernetes source checkout for the kubelet binary, neither of which contains it on a prebuilt image. Pass --k8s-bin-dir=$(dirname $KUBELET_BIN) so the framework execs the version-matched kubelet that's already on PATH. 2. klog.Fatalf exits with 255 — the same code ssh uses for its own transport/auth failures. Add an EXIT trap in the helper that translates remote exit 255 -> 254, so the workflow can distinguish conformance failures from SSH transport problems. 3. The workflow exited on the suspected SSH-transport path before trying to scp results back, which destroyed the partial logs that would have made triage easier. Move the results-scp ahead of the exit-decision so we always attempt to retrieve whatever was produced before reporting failure.
First successful run produced 4 pass / 19 fail / 742 skip over 2h with kubelet.log showing 1,425 copies of: Container runtime network not ready ... cni plugin not initialized The node never reaches Ready, so pods do not schedule and almost every NodeConformance test times out waiting on its pod. CAPI images ship CNI plugin binaries under /opt/cni/bin but no CNI config under /etc/cni/net.d — that file is normally written by the CNI provider on cluster join. For standalone node-conformance there's no provider, so write a minimal bridge + loopback config ourselves before launching e2e_node.test. Pre-existing CNI config is left alone so we don't conflict with a real CNI install. Plugin binaries are downloaded from containernetworking/plugins only when missing from the image.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status: WIP / Draft — opening for iteration and review.
What this does
Adds an opt-in, non-blocking step to the
Build Azure SIG Imageworkflow thatruns the [
[NodeConformance]] subset of Kubernetes'e2e_nodetest suiteagainst the freshly-baked SIG image's test VM. The goal is to give us real
signal that the produced node image actually behaves like a conformant
Kubernetes node, beyond the existing version/smoke checks.
Changes
run_node_conformance(defaultfalse).images/capi/hack/node-conformance.shthat runs on thetest VM:
kubernetes-test-linux-${ARCH}.tar.gzfrom
dl.k8s.io(rather than the staleregistry.k8s.io/node-test:0.2).kubeletso the framework can start its own under-testkubelet.
e2e_node.test --standalone-mode=trueagainst the system containerdsocket, focused on
[NodeConformance], skipping[Flaky]|[Serial]|[Slow]|[Alpha]./tmp/node-conformance-results/.e2e subset requires a real cluster).
continue-on-error: true— failures surface as a workflow warning butdon't block the promote stage while we iterate.
node-conformance-resultsis uploaded as an artifact.timeout-minutesfor thetestjob is conditionally bumped to150only when conformance is enabled; default
15is preserved otherwise.Why opt-in / non-blocking initially
registry.k8s.io/node-test:0.2image is stale; we're using theversioned binary tarball from
dl.k8s.ioinstead, but the standalone-modeinvocation arguments may need tuning per OS (Ubuntu vs AzureLinux, cgroup
paths, hugepages, AppArmor, etc.).
Azure VMs. We want signal, not a broken pipeline, while we triage.
opt-in lets us run it on demand from the workflow dispatch UI.
Open questions / follow-ups
--kubelet-flagsper OS distro (current values are conservativedefaults aimed at containerd + systemd cgroup driver).
Standard_D2s_v3when conformanceis enabled — some tests are memory-hungry.
blocking (default-on) for promote.
GINKGO_SKIPset.Testing
shellcheck -x.workflow with
run_node_conformance: trueand iterate from real output.