k8s: support Sysbox-CE on Flatcar 4593+ (kernel 6.x)#153
Open
Till0196 wants to merge 4 commits into
Open
Conversation
k3s and RKE2 with containerd 2.x regenerate /etc/containerd/config.toml on every restart, so the existing dasel-based config.toml editing path is wiped out on the next reboot. The supported extension point is the config-v3 drop-in directory (config-v3.toml.d), which is read after the generated base config and merged on top of it. Detect k3s / RKE2 via systemd, resolve the per-distro drop-in dir under /var/lib/rancher/<dist>/agent/etc/containerd/, and write a minimal sysbox-runc runtime block as a standalone drop-in. The drop-in body is shipped as a template (config/containerd-sysbox- dropin.toml.tmpl) and rendered with a sed substitution at install time so the TOML lives next to the other k8s config artifacts rather than embedded in the deploy script. Vanilla containerd nodes keep their existing config.toml-editing path. unconfig_containerd_for_sysbox is updated to mirror the new path: on k3s / RKE2 it simply removes the drop-in and restarts the wrapper service.
The drop-in only ever substituted the sysbox-runc binary path. Carrying it as a .tmpl with an @SYSBOX_RUNC_PATH@ placeholder is heavier than needed: rename it to a plain .toml that is valid on its own and hard-code BinaryName to /usr/bin/sysbox-runc (the only path the installer ever writes on non-Flatcar nodes). Simplify write_containerd_sysbox_dropin() to a straight cp. config_containerd_for_sysbox() still computes sysbox_runc_path for the legacy dasel-based containerd.conf branch.
Flatcar Container Linux 4593 and later ship a 6.x kernel where the sysbox-flatcar-preview shiftfs.ko no longer applies; the kernel provides idmap mounts instead, which sysbox-runc uses transparently. Mirror the non-Flatcar branch's kernel-version gate in install_sysbox_deps so shiftfs installation is skipped above the shiftfs_max_kernel_ver. Also fall back to the generic binary directory for Sysbox-CE on Flatcar, since the per-release flatcar-* artifact dir only exists in the EE build that depends on shiftfs.
Drop the EE-only guard in do_distro_adjustments(). The guard was added when shiftfs was the only path to make sysbox work on Flatcar, and shiftfs was an EE-exclusive preview build. On kernel 6.x Flatcar ships mainline idmap mount support, which sysbox-runc uses transparently; the EE-only justification no longer holds, and the companion change to skip install_sysbox_deps_flatcar() above the shiftfs ceiling already makes the install path correct for CE on 6.x. Also fold the Flatcar-specific BinaryName rewrite for the containerd drop-in into do_distro_adjustments(), matching how every other Flatcar path override (systemd units, crio.conf, helper scripts) is handled in the same function. Addresses: nestybox/sysbox#995
There was a problem hiding this comment.
Pull request overview
Enables Sysbox-CE installs on Flatcar 4593+ (kernel 6.x) by removing the EE-only guard in the Flatcar install path and by routing the new containerd drop-in (introduced for k3s/RKE2 in #152) through the existing Flatcar path-rewrite logic so BinaryName lands at /opt/bin/sysbox-runc. On 6.x kernels the shiftfs-preview install is skipped because the prebuilt module no longer applies and idmap mounts cover the same use cases.
Changes:
get_artifacts_dir: route Sysbox-CE on Flatcar tobin/generic(no Flatcar-specific shiftfs build is shipped for CE).install_sysbox_deps: on Flatcar, only callinstall_sysbox_deps_flatcarwhen kernel < 6.3; otherwise skip with a clear message.do_distro_adjustments: drop the EE-onlydie, andsed-rewritecontainerd-sysbox-dropin.toml'sBinaryNameto/opt/bin/sysbox-runc(the CE Dockerfile and the containerd drop-in/loader logic from #152 are also included in this diff).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| k8s/scripts/sysbox-deploy-k8s.sh | Flatcar CE artifact routing, kernel-gated shiftfs install, EE-only guard removal, Flatcar path rewrite for the containerd drop-in, and (from #152) k3s/RKE2 drop-in install/uninstall path. |
| k8s/Dockerfile.sysbox-ce | Copies the new containerd-sysbox-dropin.toml into the daemonset image. |
| k8s/config/containerd-sysbox-dropin.toml | New static drop-in registering sysbox-runc under the containerd v3 plugin key. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # kernel provides idmap mounts), so the prebuilt shiftfs.ko from | ||
| # sysbox-flatcar-preview no longer applies. | ||
| if semver_lt $kversion 6.3; then | ||
| install_sysbox_deps_flatcar |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the Flatcar-specific follow-up to #152 (containerd drop-in for k3s / RKE2). On a 6.x Flatcar node running RKE2, the install dies at the EE-only guard in
do_distro_adjustments()before #152's drop-in logic gets a chance to run.The historical reason for that guard was shiftfs being EE-exclusive (the
sysbox-flatcar-previewbuild), but on 6.x:sysbox-flatcar-previewshiftfs.kono longer applies (it isn't maintained for the 6.x line; module load fails).idmapmounts in mainline, which sysbox-runc uses transparently in place of shiftfs.So shiftfs isn't reachable on 6.x even if you wanted it, and the alternative (
idmap) isn't EE-restricted. This PR tries to make CE on 6.x Flatcar work:install_sysbox_deps_flatcaron kernels ≥ 6.x — the function only installs the shiftfs preview, which can't build on 6.x. Fall back tobin/genericartifacts.BinaryNameto/opt/bin/sysbox-runcon Flatcar (Flatcar's/usris read-only). This piggybacks on the existing Flatcar path-rewrite sed pile indo_distro_adjustments().dieblock is taken out — see the paragraph below; if the policy still applies for reasons other than shiftfs availability, this is the bullet to flag and I'll drop the change.I don't have full context on whether the "Flatcar = EE only" policy from PR #48 (2021-09) still applies for non-shiftfs reasons (licensing, support scope, etc.). If there's something I'm missing, happy to adjust or drop the relevant parts — the verification below is just to show that the technical pieces line up.
Verification
Tested on Flatcar 4593.2.1 + kernel 6.12.87 + RKE2 v1.36.0+rke2r1 + containerd 2.2.3-k3s1 + Cilium CNI, with stock upstream sysbox-runc master. A statically-built
fusermount3was placed on the host at/opt/bin/fusermount3out-of-band (built from Till0196/fusermount3-static; see the fusermount note below for why):dieguard)runtimeClassName: sysbox-runc+hostUsers: false(alpine)runtimeClassName: sysbox-runcon EE × Flatcar 4593+ (kernel 6.x)Open items / known limitations
fusermount3still needs to be reachable on the host. sysbox-fs in the v0.7.0 deb invokesfusermount3to set up its FUSE mounts, and Flatcar's base image doesn't ship one. Until nowinstall_sysbox_deps_flatcarsupplied it alongside the shiftfs preview; skipping that function on 6.x removes that path.For the verification above I built a statically-linked
fusermount3(Till0196/fusermount3-static — a small wrapper around libfuse'sutil/fusermount.cfor minimal distros like Flatcar / Talos / distroless) and dropped it into/opt/bin/out-of-band. That's enough to make sysbox-fs work.One option going forward is for the daemonset to bundle a small static
fusermount3independently of the shiftfs flow. I left it out of this PR because adding a binary the project then has to maintain feels like a maintenance-cost call that should be on your side rather than mine.As a separate, parallel attempt, nestybox/fuse#10 proposes letting sysbox-fs call mount(2) directly when running as root, so the
fusermount3binary isn't needed at all. I also ran the FUSE-touching scenarios above with a sysbox-fs built against that branch and nofusermount3on the host — they passed.If Flatcar support is going to continue, my impression is that the fuse-side route is the cheaper integration overall: a small library patch, with no extra static binary to build, ship, and keep in sync with the deb. So between the two options above, I'd softly lean toward picking up the fuse PR rather than adding a static
fusermount3to the daemonset — but that's a judgment call from your side.K8s version: tested on v1.36, which is currently outside
is_supported_k8s_version's allowlist. Allowlist extension is tracked in Kubernetes 1.33 support sysbox#961, separately.