Livepatch 6.12.95 v6: stalled transition leaves nf_tables unregistered (nft returns EINVAL in containers)

[jiri001meitner]

Summary

On a vpsAdminOS node running 6.12.95 with livepatch replacement v4 → v6, the
transition of livepatch_6 has not completed. As long as it stays in
transition, nf_tables is unavailable inside the container: every nf_tables
netlink request fails with EINVAL. Other nfnetlink subsystems (conntrack) and
rtnetlink keep working.

Another VPS of ours on a different node completed the same transition and works
normally.

Observed state (inside the affected VPS)

$ uname -r
6.12.95.4
$ cat /sys/kernel/livepatch/livepatch_6/{enabled,transition,replace}
1
1
1

Working VPS for comparison: uname -r = 6.12.95.6, transition = 0.

$ nft list tables
netlink: Error: cache initialization failed: Invalid argument
$ unshare -n nft list tables          # fresh, empty netns
netlink: Error: cache initialization failed: Invalid argument
$ iptables -S                          # nf_tables backend
iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Invalid argument
$ conntrack -C
1361

strace nft list tables shows the kernel answering NFT_MSG_GETGEN and
NFT_MSG_GETTABLE with NLMSG_ERROR / -EINVAL.

Timeline: an hourly job using nft last succeeded at 2026-09-14 03:31:55 CEST
and has failed since 04:34:18 CEST. Nothing changed inside the VPS in that
window (no package updates, service restarts or container events), and dmesg
is clean. No task inside the VPS is sleeping in nfnetlink, nf_tables or
livepatch code, so the task that blocks the transition is presumably on the
host or in another container on the node.

Cause in the source

In os/livepatches/bp-6.12.95-cumulative.patch (v6, d9bd139cd):

• the pre_patch callback vpsadminos_nftables_livepatch_quiesce() calls
  vpsadminos_nfnl_try_unregister(&nf_tables_subsys), which clears
  table[NFNL_SUBSYS_NFTABLES].subsys;
• the subsystem is registered again only by
  vpsadminos_nftables_livepatch_restore(), called from the post_patch (or
  post_unpatch) callback.

While the transition is pending, nfnetlink_rcv_msg() finds no subsystem for
NFNL_SUBSYS_NFTABLES and returns -EINVAL, which matches the behaviour
above. The quiesce window is therefore unbounded: if the transition never
completes, nf_tables stays unreachable for every container on the node.

Already-loaded rulesets appear to remain in place (only the netlink subsystem
is unregistered), so existing filtering should still apply, but nothing can
list or change the rules.

Impact

• Firewall rules cannot be listed, validated or changed in any container on the
  node (nft, iptables-nft, configuration management using nft --check).
• Restarting nftables.service or rebooting the VPS in this state would flush
  the rules and fail to load new ones, leaving the container unfiltered.

The same pattern applies to ip_set_netlink_subsys and nfqnl_subsys, which
are quiesced by the same livepatch and restored only from post_patch as well.
The quiesce code has been present since v4 (89773cc3a), so this is not a v6
regression; the transition simply stalled this time.

Immediate remedy for the affected node

Complete or reverse the stalled transition. Reversal (echo 0 > enabled) keeps
the previous patch active thanks to atomic replace and, once the reverse
transition completes, calls post_unpatch, which restores the quiesced
subsystems.

Proposed fix

1. live-patches load should not leave the transition unattended. Today
   moduleLoadGen runs insmod, checks enabled and records applied-at
   even while transition is still 1. Proposal:

   • wait for transition to reach 0 with a configurable bound
     (services.live-patches.transitionTimeout, e.g. 300 s; the kernel already
     sends fake signals every 15 s, so healthy transitions finish much sooner),
   • on expiry, log the tasks still unpatched
     (/proc/*/task/*/patch_state = 0, with their stacks) and reverse the
     transition, waiting for the reversal as well,
   • record applied-at only after the transition has completed.

   I have a draft of this change for os/modules/services/livepatches/default.nix
   and can open it as a PR, but I have no way to run the livepatch test suite
   myself. A test could reuse the existing kprobe holds in
   tests/suite/kernel/livepatch-6.12.95.nix to stall the transition and then
   assert that it is reversed and that nft list tables / ipset list -n work
   again.

2. Kernel side (for discussion). Quiescing a subsystem for the whole
   duration of a node-wide transition ties its availability to unrelated
   blocking tasks (FUSE, NFS, KVM, ...). Options:

   • while quiesced, register a stub subsystem that answers with -EBUSY and
     an extack message ("nf_tables unavailable: livepatch transition in
     progress") instead of leaving the slot empty (-EINVAL), so the state is
     diagnosable from inside a container;
   • restrict the quiesce to the part of the transition where old and new
     nf_tables code could actually interleave, if that can be determined.

Happy to adjust the proposal to whatever direction you prefer.

Happy to provide more details privately (node / VPS identifiers) if needed.

[snajpa]

not a bug in any stable release, normal part of development