k8s: ship sysbox-runc as a containerd drop-in for k3s / RKE2#152
Open
Till0196 wants to merge 2 commits into
Open
k8s: ship sysbox-runc as a containerd drop-in for k3s / RKE2#152Till0196 wants to merge 2 commits into
Till0196 wants to merge 2 commits into
Conversation
k3s and RKE2 with containerd 2.x regenerate /etc/containerd/config.toml on every restart, so the existing dasel-based config.toml editing path is wiped out on the next reboot. The supported extension point is the config-v3 drop-in directory (config-v3.toml.d), which is read after the generated base config and merged on top of it. Detect k3s / RKE2 via systemd, resolve the per-distro drop-in dir under /var/lib/rancher/<dist>/agent/etc/containerd/, and write a minimal sysbox-runc runtime block as a standalone drop-in. The drop-in body is shipped as a template (config/containerd-sysbox- dropin.toml.tmpl) and rendered with a sed substitution at install time so the TOML lives next to the other k8s config artifacts rather than embedded in the deploy script. Vanilla containerd nodes keep their existing config.toml-editing path. unconfig_containerd_for_sysbox is updated to mirror the new path: on k3s / RKE2 it simply removes the drop-in and restarts the wrapper service.
The drop-in only ever substituted the sysbox-runc binary path. Carrying it as a .tmpl with an @SYSBOX_RUNC_PATH@ placeholder is heavier than needed: rename it to a plain .toml that is valid on its own and hard-code BinaryName to /usr/bin/sysbox-runc (the only path the installer ever writes on non-Flatcar nodes). Simplify write_containerd_sysbox_dropin() to a straight cp. config_containerd_for_sysbox() still computes sysbox_runc_path for the legacy dasel-based containerd.conf branch.
There was a problem hiding this comment.
Pull request overview
Updates the Kubernetes deploy script to register sysbox-runc with containerd on k3s/RKE2 using the supported config-v3.toml.d drop-in mechanism (instead of editing a regenerated config.toml), so runtime registration persists across agent restarts.
Changes:
- Detect k3s/RKE2 via systemd and write a containerd
config-v3drop-in at the Rancher-managed path. - Add a helper to restart the appropriate owning service (k3s/rke2 vs. vanilla containerd) after config changes.
- Ship a static TOML artifact in the sysbox-ce image for the drop-in content.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| k8s/scripts/sysbox-deploy-k8s.sh | Adds k3s/RKE2 detection, drop-in write/remove logic, and service restart dispatch. |
| k8s/Dockerfile.sysbox-ce | Includes the new containerd drop-in artifact in the image. |
| k8s/config/containerd-sysbox-dropin.toml | Defines the sysbox-runc runtime block for containerd config-v3 drop-in usage. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+42
to
+45
| # Containerd drop-in used on k3s / RKE2. Ships with /usr/bin/sysbox-runc; on | ||
| # Flatcar do_distro_adjustments() rewrites this artifact to /opt/bin/sysbox-runc | ||
| # up-front, so the install path can copy it verbatim. | ||
| containerd_sysbox_dropin_src="${sysbox_artifacts}/config/containerd-sysbox-dropin.toml" |
| [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.sysbox-runc] | ||
| runtime_type = "io.containerd.runc.v2" | ||
| [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.sysbox-runc.options] | ||
| BinaryName = "/usr/bin/sysbox-runc" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Sysbox v0.7.0 deb already ships everything needed to run sysbox-runc pods with
hostUsers: falseon K8s + containerd v2 -- the maintainer confirmed as much in nestybox/sysbox-runc#109 ("Sysbox v0.7.0 supports K8s pods with user-namespaces"). Verified empirically: the v0.7.0 deb'ssysbox-runc(commita4dd414f) carries thefeaturessubcommand from sysbox-runc#106, which is what advertises userns support to containerd 2.x's runtime introspection (internal/cri/server/service.go::supportsCRIUserns).On k3s / RKE2 the deploy script cannot actually land that binary stack though, because the
dasel-based edit of/etc/containerd/config.tomlis wiped on every agent restart (k3s and RKE2 regenerate the file from a template). This PR writes the sysbox-runc runtime block as aconfig-v3.toml.ddrop-in instead -- the supported extension point on those distros.Problem
sysbox-deploy-k8sfails on k3s / RKE2 with containerd v2 (sysbox#1000):Even when the file is created, k3s/RKE2 regenerate it on the next agent restart so any edits are dropped silently -- the user's runtime config disappears between reboots.
Fix
k8s_dist_owning_containerd).sysbox.tomlto the distro'sconfig-v3.toml.ddirectory:/var/lib/rancher/<dist>/agent/etc/containerd/config-v3.toml.d/sysbox.toml.k8s/config/*artifacts.BinaryNameis hard-coded to/usr/bin/sysbox-runc(the only path the installer ever writes for sysbox-CE on non-Flatcar nodes).dasel-based edit path unchanged.Why drop-in, not path redirect
host_containerd_conf_fileto/var/lib/rancher/<dist>/.../config.tomlconfig-v3.toml.d(this PR)Note on
SupportsUsernsThis PR does not set
SupportsUserns = trueon the runtime entry. The key is not a field on containerd'sRuntimeconfig struct (internal/cri/config/config.go) and is silently discarded by the TOML decoder -- empirically verified on containerd 2.2.3-k3s1 (no behavioural difference with or without the line; net-ns ownership and OCI specuser.pathare identical). The real userns gating mechanism is the<runtime> featuressubcommand result, which sysbox-runc already handles since nestybox/sysbox-runc#106 and which is present in the v0.7.0 deb.Test plan
Verified on Flatcar 4593.2.1 + RKE2 v1.36.0+rke2r1 + containerd 2.2.3-k3s1 + Cilium CNI, with stock v0.7.0 deb binaries (no out-of-tree patches, extracted directly from the sysbox-deploy-k8s container image: sysbox-runc commit
a4dd414f, sysbox-fs commitb70bd38b, sysbox-mgr commitaa8f237c):/var/lib/rancher/rke2/agent/etc/containerd/config-v3.toml.d/sysbox.toml; survivesrke2-agentrestartruntimeClassName: sysbox-runc+hostUsers: false(alpine sleep)uid_mapshows K8s-allocated level-1 range; sysfs/proc mounted; sysbox-fs FUSE serves/proc/sys; binfmt_misc visibleruntimeClassName: sysbox-runc+hostUsers: false+ docker:dindalpine:3.19, inner container gets bridge IP, outbound to1.1.1.1succeedsThis matches the maintainer's claim in nestybox/sysbox-runc#109 that v0.7.0 supports
hostUsers: falsepods -- but only once the deploy script actually lands the runtime registration. This PR is what makes that happen on k3s / RKE2.The vanilla containerd dasel path was not exercised in the manual test; please flag if additional verification is wanted.
Scope
hostUsers: true(the Pod-spec default) still fails at sysfs EPERM on this stack with any sysbox-runc version. The maintainer's position in feat: Add external user namespace support sysbox-runc#109 is thathostUsers: falseis the supported path; this PR follows that direction.