Windows: kubelet installed via sc.exe (windows_service_manager=windows_service) ignores KUBELET_KUBEADM_ARGS, breaking Cluster API kubeletExtraArgs

[mboersma]

What happened

Windows images built with windows_service_manager: windows_service (the sc.exe path added in commit f7d8f02) install the kubelet service with a static binPath that does not consume /var/lib/kubelet/kubeadm-flags.env. As a result, every kubelet argument that Cluster API / kubeadm passes via nodeRegistration.kubeletExtraArgs is silently dropped at kubelet startup.

The current images/capi/ansible/windows/roles/kubernetes/tasks/sc.yml even calls this out in a comment:

Does not support kubeadm KUBELET_KUBEADM_ARGS which is used by Cluster API to pass extra user args

…but in practice this isn't a workaround that consumers can opt into — it silently breaks any Cluster API–driven Windows workload that relies on kubeletExtraArgs. The Linux equivalent (systemd) reads EnvironmentFile=/var/lib/kubelet/kubeadm-flags.env, so Linux and Windows nodes built from these images now have divergent semantics for the same Cluster API config.

Concrete impact (observed)

In the cloud-provider-azure-ccm-windows-capz job, using capi-win-2022-containerd:latest from the CAPZ CI gallery:

• The KubeadmConfig has kubeletExtraArgs.cloud-provider=external, --register-with-taints=node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule, --v=2, --windows-priorityclass=ABOVE_NORMAL_PRIORITY_CLASS, etc.
• kubeadm join runs cleanly, writes kubeadm-flags.env, starts the kubelet service.
• The Windows kubelet service launched via sc.exe ignores all of the above — no --cloud-provider=external, no --register-with-taints.
• Result: the Windows Node registers with only node.kubernetes.io/not-ready:NoSchedule, no node.cloudprovider.kubernetes.io/uninitialized taint.
• cloud-node-manager-windows logs Node has no cloud taint, skipping initialization. spec.providerID is never set. Cloud Controller Manager E2E and conformance jobs for Windows fail.

Linux nodes in the same run are unaffected — they get the uninitialized taint (auto-applied by kubelet from --cloud-provider=external) and CNM initializes them normally.

A separate Cluster API Provider Azure workaround (CAPZ #6293) tried to make the taint explicit via kubeletExtraArgs.register-with-taints; it had no effect, because kubeletExtraArgs itself is what's being dropped.

Root cause

sc.yml installs the service with:

path: >-
  "{{ kubernetes_install_path }}\kube-log-runner.exe" --log-file=…\var\log\kubelet\kubelet.log
  {{ kubernetes_install_path }}\kubelet.exe --windows-service
  --cert-dir=… --config=…\config.yaml --bootstrap-kubeconfig=… --kubeconfig=…
  --enable-debugging-handlers --cgroups-per-qos=false --enforce-node-allocatable=""
  --container-runtime-endpoint="npipe:////./pipe/containerd-containerd"
  --resolv-conf=""

kube-log-runner is a stdout/stderr redirector; it does not read env files. So KUBELET_KUBEADM_ARGS from /var/lib/kubelet/kubeadm-flags.env is never appended to the kubelet argv. By contrast, the nssm.yml path runs templates/StartKubelet.ps1, which does read that file.

Proposed fix

Make the sc.exe service launch through the same StartKubelet.ps1 wrapper that the nssm path already uses, so both flows consume kubeadm-flags.env identically. Roughly:

- name: Render StartKubelet wrapper
  ansible.windows.win_template:
    src: templates/StartKubelet.ps1
    dest: "{{ kubernetes_install_path }}\\StartKubelet.ps1"

- name: Install kubelet as service
  ansible.windows.win_service:
    name: kubelet
    start_mode: auto
    path: >-
      "{{ kubernetes_install_path }}\kube-log-runner.exe"
      --log-file={{ systemdrive.stdout | trim }}/var/log/kubelet/kubelet.log
      powershell.exe -ExecutionPolicy Bypass -NonInteractive
      -File "{{ kubernetes_install_path }}\StartKubelet.ps1"

kubelet --windows-service should be dropped from the binPath since kubelet is now a child of the wrapper rather than the service entry point.

#2009 hardens StartKubelet.ps1 and should land first, so both flows benefit from the same fixes (substring strip of KUBELET_KUBEADM_ARGS=, splatting instead of Invoke-Expression, optional start-kubelet log).

Alternatives considered

• Add --env-file=… to upstream kube-log-runner to splat env-file values onto its child's argv. Cleaner long-term (no PowerShell-as-service), but a Kubernetes-side change with a longer release path.
• Have CAPZ paper over this on the consumer side (e.g., post-join sc.exe config kubelet binPath= …). Possible, but pushes a Windows packaging responsibility into every Cluster API provider.

Environment

• Image: capi-win-2022-containerd (Windows Server 2022, containerd) from CAPZ's CI Compute Gallery ClusterAPI-f72ceb4f-5159-4c26-a0fe-2ea738f0d019.
• Built with windows_service_manager: windows_service (sc.exe path enabled by f7d8f02).
• kubelet binary: replaced at boot by CAPZ's replace-ci-binaries.ps1 with k8s latest CI build (currently v1.37.0-alpha.0.750+17274240a1d95e). The same bug reproduces with the image's baked kubelet; the version isn't the cause.

/area provider/azure
/kind bug