nut-client: extension thrashes on FSD instead of powering off host

[jhogendorn]

Summary

The nut-client extension's SHUTDOWNCMD lifecycle interacts badly with the extension's restart: always policy: when the NUT primary broadcasts FSD, the extension's upsmon receives it, runs SHUTDOWNCMD, exits cleanly — and containerd immediately restarts it. The next poll still sees FSD on upsd, so the cycle repeats roughly every 5 seconds (the upsmon POLLFREQ default). The Talos node never powers off. The thrash continues until the FSD flag is cleared on the primary's upsd (e.g. operator restarts the primary's upsd).

This means the extension does not deliver host shutdown in the canonical NUT-secondary scenario it advertises.

Environment

• Talos: v1.12.5 (latest released v1.13.2; lag is not material to this report)
• Extension: nut-client carrying NUT 2.8.4 per talosctl get extensionstatus. Upstream power/vars.yaml currently pins NUT_VERSION: 2.8.5 — the +0.0.1 delta does not affect the lifecycle described here.
• Schematic ID: a1112c2adc28523383a96a65e2ce7a4d1adfa3fb2ced4bed26e2ca4a2ade9a80
• Primary: NUT 2.8.0 (Debian 12) on a separate host; UPS via SNMPv1 to an APC NMC2
• 3 worker nodes installed with the extension; all observe the same behaviour

The structural bug is present on main HEAD as of the time of this report: power/nut-client/nut-client.yaml still sets restart: always, and there have been no commits to power/nut-client/ since 2026-04-10 (a dependency bump). Happy to re-run the repro on v1.13.2 if the maintainers prefer; the lifecycle described here is at the extension+containerd layer and does not depend on the Talos kernel version.

Repro

1. Install the nut-client extension on a Talos node.

2. Apply an ExtensionServiceConfig with the standard secondary upsmon.conf:

   apiVersion: v1alpha1
   kind: ExtensionServiceConfig
   name: nut-client
   configFiles:
     - content: |-
         MONITOR <ups>@<primary>:3493 1 <user> <pass> secondary
         SHUTDOWNCMD "/sbin/poweroff"
       mountPath: /usr/local/etc/nut/upsmon.conf

3. From the NUT primary on another host, issue an FSD broadcast:

   upsmon -c fsd

4. Observe: talosctl logs ext-nut-client shows the FSD handling loop. talosctl service ext-nut-client shows rapid restart cadence. The host never powers off.

5. Clear the FSD flag on the primary's upsd (restart nut-server) to stop the thrash.

Observed behaviour

talosctl logs ext-nut-client:

UPS rack-ups-01@<primary>:3493: forced shutdown in progress
Executing automatic power-fail shutdown
Can't invoke wall: No such file or directory
Auto logout and shutdown proceeding
Network UPS Tools upsmon 2.8.4 release
UPS: rack-ups-01@<primary>:3493 (secondary) (power value 1)
No POWERDOWNFLAG value was configured in /usr/local/etc/nut/upsmon.conf!
POWERDOWNFLAG should be a path to file that is normally writeable for root user, ...
Warning: no custom notification command defined, just so you know
upsnotify: failed to notify about state NOTIFY_STATE_READY_WITH_PID: no notification tech defined, ...
UPS rack-ups-01@<primary>:3493: forced shutdown in progress
Executing automatic power-fail shutdown
...

talosctl service ext-nut-client (excerpt):

EVENTS   [Running]: Started task ext-nut-client (PID 81369) for container ext-nut-client (56s ago)
         [Waiting]: Runner Containerd(ext-nut-client) exited without error, going to restart it (1m1s ago)
         [Running]: Started task ext-nut-client (PID 81126) for container ext-nut-client (1m6s ago)
         [Waiting]: Runner Containerd(ext-nut-client) exited without error, going to restart it (1m11s ago)
         [Running]: Started task ext-nut-client (PID 80900) for container ext-nut-client (1m16s ago)
         ...

12+ restart/exit cycles per minute. The pattern continues until the FSD flag is removed from upsd.

The primary's upsd logs confirm the secondary connected and read ups.status "FSD OL" before each restart — so the FSD broadcast is correctly delivered and observed.

Diagnosis

Three things compose to produce the loop:

1. upsmon exits cleanly after running SHUTDOWNCMD. This is normal NUT secondary behaviour: the binary's job is finished once it has invoked the configured shutdown command.
2. The extension's containerd restart policy is restart: always. When upsmon exits with rc=0, containerd re-launches it.
3. The primary's upsd retains the FSD flag on ups.status until explicitly cleared (e.g. UPS service / restart of nut-server). This is correct NUT primary behaviour — FSD is "the shutdown is committed, do not retract".

So every restart of the extension repeats the SHUTDOWNCMD invocation. The /sbin/init-bind-mounted-to-/sbin/poweroff trick (clever — machined dispatches on argv[0]) gets cut off before machined can complete its host-shutdown sequence, because the next FSD invocation reaches the extension container before machined has finished the previous one.

There is no community report of this issue in siderolabs/extensions — searches for "nut", "fsd", "shutdown", "poweroff" return zero matches. This suggests the FSD path is not commonly exercised end-to-end in production; configurations are typically validated by deploying the extension and confirming it polls, without firing a real FSD.

Proposed fixes

Three shapes, in order of minimal-change:

Option 1 — change restart policy to on-failure

Single line change in power/nut-client/nut-client.yaml:

restart: on-failure

Pros: minimal patch, intent is clear (a clean upsmon exit is "job done, don't restart"; real crashes still restart). No new code paths.
Cons: subtle behavioural change if upsmon exits cleanly for non-FSD reasons (e.g. upsmon -c stop from outside, which doesn't happen in this setup).

Option 2 — POWERDOWNFLAG-aware entrypoint wrapper

Add POWERDOWNFLAG /etc/nut/killpower to the bundled upsmon.conf. Add a small entrypoint script: if upsmon exits 0 AND /etc/nut/killpower exists, sleep infinity (signals "host shutdown is in flight; please do not restart"). Otherwise exit non-zero to let containerd's restart policy take effect.

Pros: uses NUT's documented mechanism. Works with any restart policy. Robust to surprises.
Cons: requires entrypoint script + slightly more surface area.

Option 3 — combine the above

restart: on-failure + bundled POWERDOWNFLAG set in upsmon.conf so operators get the canonical NUT lifecycle for free.

---

Happy to send a PR for whichever shape you prefer. Documenting here first because the bind-mount trick suggests there was deliberate design thinking around the host-shutdown path, and I'd rather defer to your preferred restart-suppression model than guess.

Workaround for operators hitting this today

External orchestration: drive Talos shutdowns from the NUT primary via the Talos API instead of relying on the extension. On the primary host:

talosctl shutdown --nodes <worker-1>,<worker-2>,...

invoked from the primary's FSD path (e.g. as a custom NOTIFYCMD for the FSD event in the primary's upsmon, or from a daemon that watches upsd state). Requires talosctl + a valid talosconfig on the primary host. Strip the nut-client extension from worker schematics in this model — it's a single point of orchestration outside the cluster, which is the right model for power-event coordination on an API-driven OS.

[frezbo]

Please attach a support bundle of the node, do you have talos console logs to prove talos did not shutdown, keeping aside the fact that the extension restarts, once talos receives a shutdown requests it should shutdown even if the service restarts, so there must be something else that prevents shutdown

[jhogendorn]

Correction — host does eventually power off, with significant delay

After filing this issue I let the cluster sit, then re-checked an hour
later and found all three worker VMs in stopped state on the
hypervisor. The original framing — "the Talos node never powers off"
— is wrong. The host does power off, but with characteristics
that may be worse than the simple "never powers off" failure mode I
originally described.

Refined timeline (single FSD test, 3 worker nodes)

T+0:00  upsmon -c fsd on primary; upsd flips ups.status to "FSD OL"
T+0:03  ext-nut-client on all 3 workers logs "forced shutdown in progress"
        → "Executing automatic power-fail shutdown"
        → "Auto logout and shutdown proceeding"
        Containerd restart loop begins, ~12 cycles/min.
T+~1:00 Operator clears FSD flag on the primary (restart of nut-server).
        Extension stops thrashing — next poll sees plain "OL" again.
T+~7:00 hypervisor qmeventd records VM termination for all 3 workers
        (within a 7-second window: 16:25:42 / :45 / :49 for the 3
        node VMIDs).

So:

• The host shutdown commits ~7 minutes after the FSD broadcast.
• The host shutdown commits ~6 minutes after the FSD flag has been
  cleared on the primary. From an operator's perspective: triggering
  FSD and quickly clearing it does not retract the in-flight shutdown.
• During the thrash window, talosctl calls succeed; kubelet is still
  serving. Then the node disappears.

Why this is arguably worse than "never shuts down"

A "never shuts down" failure is loud: the operator notices the host is
still up and intervenes. The actual behaviour is silent and delayed —
the orderly shutdown does land, just on an unpredictable timeline
the operator cannot predict or cancel. An accidental upsmon -c fsd
(or any transient FSD flag) commits a node shutdown several minutes
in the future, regardless of what state the primary is in when the
shutdown actually triggers.

What I think is happening

The /sbin/init-as-/sbin/poweroff bind mount + machined argv[0]
dispatch design works: each cycle of the thrash sends a poweroff
request to machined via the bind-mounted socket. machined queues
the shutdown, runs the standard sequence (cordon, drain, kill kubelet,
stop services, unmount, poweroff). The sequence takes time
proportional to running workload. Subsequent thrash-cycle invocations
arrive while a shutdown is already in flight; they likely become
no-ops or are coalesced. Eventually the sequence completes and the
VM exits.

The restart: always policy doesn't prevent shutdown — it just
makes the shutdown timing depend on however long it takes machined
to walk the full sequence with the workload that was running.

Implication for the proposed fixes

All three fixes I proposed earlier still apply, and the case for them
gets stronger:

• Option 1 (restart: on-failure). Clean exit from upsmon after
  the first poweroff invocation → extension stays exited → machined
  gets a single, deterministic shutdown request to process.
• Option 2 (POWERDOWNFLAG-aware entrypoint). Same outcome, more
  defensive — distinguishes intentional clean exit from a real crash.
• Option 3 (combine). Belt and braces.

Net effect either way: the host shutdown still happens (good), but
without the thrash storm of restart cycles spamming the journal and
keeping machined busy. The timing becomes "however long the standard
machined shutdown takes" instead of "however long machined takes
while also fielding 12 restart-init cycles per minute".

What I'll update in our internal docs

We're keeping the strip-the-extension workaround in our infra repo
(external talosctl shutdown from the NUT primary host), because
unpredictable-timing-with-no-cancel is still untenable for our power
event runbook. But the framing of the upstream bug deserves to be
correct.

Apologies for the misleading original framing.

[frezbo]

Please drop the detailed AI slop and attach proper logs and support bundle, to confirm if service restarts affects shutdown timeout

[jhogendorn]

I reran through the scenario to produce good artifacts, it appears to be that while FSD is active, the nodes thrash. once FSD is cleared it appears to stop the looping behaviour and allow the shutdown to happen. apologies for the sloppy reply, cheeky bastard decided to go off on its own and post. sigh.

As requested, proper logs and the support bundle. i did go through the bundle and redact some secrets that leaked but its otherwise as created.

08-talos-support-bundle-sanitised.zip

00-pretest-health.txt
01-w-0-ext-nut-client.log
01-w-1-ext-nut-client.log
01-w-2-ext-nut-client.log
02-w-0-dmesg.log
02-w-1-dmesg.log
02-w-2-dmesg.log
03-cp-0-kubelet.log
04-proto-pve-journal.log
05-proto-kvm-journal.log
06-test-timeline.txt
07-proto-qmeventd-shutdown-events.log
README.md

[guntars-rakitko]

Independent reproduction on a different transport — confirming this is not specific to SNMP-attached UPS or Proxmox VMs.

Setup

• Talos v1.13.2, 6 bare-metal nodes (2 clusters × 3), schematic includes siderolabs/nut-client.
• NUT primary on TrueNAS Scale 25.10.3.1, NUT 2.8.4, driver usbhid-ups against an APC Smart-UPS SMT750I over USB. So: HID/USB on the primary side, not SNMP.
• Secondary upsmon.conf is the canonical shape — MONITOR apc1@<primary>:3493 1 upsmon <pwd> secondary, SHUTDOWNCMD "/sbin/poweroff", POLLFREQ 5. We had also added POWERDOWNFLAG /run/killpower after seeing the warning in the extension logs.

Drill (real utility loss — UPS unplugged from wall)

• T+0:00 — utility cut.
• T+3:14 — primary asserts LB, broadcasts FSD to upsd subscribers (confirmed in NAS logs).
• T+3:14 → T+~5min — six nodes enter the loop. talosctl service ext-nut-client shows [Running] Started task ext-nut-client … (5m47s ago) while [Waiting] Runner Containerd(ext-nut-client) exited without error, going to restart it lands roughly every 5–6 s. Same cadence on all six nodes.
• kubelet log shows repeated Finished Fail against the apiserver during the window; talosctl version still answers, node uptime keeps climbing (35h+ on the node we inspected — no real reboot).
• The thrash only stopped when the NAS itself powered off (i.e. when upsd died and the secondaries lost the FSD source). At that point the nodes finally completed their machined shutdown and powered off.

So the same lifecycle the OP described reproduces on USB-attached APC + Talos v1.13.2 + bare metal. Different UPS transport, different Talos version, different hardware class → same bug. It's structurally at the extension/containerd layer, as you already concluded.

POWERDOWNFLAG didn't help
Adding POWERDOWNFLAG /run/killpower to the secondary's upsmon.conf had no effect on the loop. That's expected: POWERDOWNFLAG is honoured by upsmon -K on the primary as part of the local poweroff sequence; on a secondary it doesn't gate SHUTDOWNCMD, and the flag file is set during the same upsmon exit path that containerd is about to restart anyway. So Option 2 / Option 3 in your write-up need the entrypoint wrapper to actually inspect the flag and sleep infinity — bare POWERDOWNFLAG in upsmon.conf is not enough on its own. Just flagging this for any operator landing on this thread thinking the warning in the logs is the actual fix.

+1 for Option 1 (restart: on-failure)
For a homelab where the FSD path is the only thing that matters, the one-line policy flip is the right minimal change. Clean upsmon exit after SHUTDOWNCMD is the documented "I'm done" signal — there's no scenario in the secondary-only nut-client extension where you'd want containerd to relaunch after a clean exit. Real crashes (bad config, network gone) still restart under on-failure, which is the only restart we'd ever want.

Happy to test a build with the one-line change against this hardware if it would help close the loop.

Operator workaround in the meantime
Until this lands upstream, we're stripping siderolabs/nut-client from the worker schematic and driving shutdown from the NAS via talosctl shutdown --force --nodes <…> from a NUT NOTIFYCMD on the primary. Same shape the OP described in the original "Workaround for operators" section.

[guntars-rakitko]

Follow-up from deeper digging on bare-metal Talos v1.13.2 (6 nodes, APC SMT750 via a TrueNAS NUT primary) — flagging a concrete limitation in the SHUTDOWNCMD execution path that blocks a fast (drain-skipping) shutdown, which is the real need in a UPS scenario:

1. SHUTDOWNCMD can't take arguments. The replace_system.patch runcmd() uses posix_spawn with the whole SHUTDOWNCMD string as a single argv[0] (the patch's own TODO: Support tokenizing cmd). So SHUTDOWNCMD "/sbin/poweroff --force" tries to spawn a file literally named /sbin/poweroff --force → ENOENT → upsmon logs Unable to call shutdown command ... (-1) and the node never powers off. Only a bare single-token path works.

2. Even if tokenized, /sbin/poweroff ignores --force. The bind-mounted /sbin/init→poweroff.Main calls client.Shutdown() with no Force option (ActionFromArgs doesn't recognize --force), so there's no way to set ShutdownRequest.Force=true — the thing that skips CordonAndDrainNode.

Why this matters: on a whole-rack power loss, all control-plane nodes go down together → etcd quorum is lost → the drain phase burns its hardcoded 5-min DrainTimeout, so each node takes ~6-8 min to power off — which can outlast UPS battery runtime. A drain-skipping shutdown (~3 min measured via talosctl shutdown --force) is essential here, but it's currently unreachable from the extension. The only fast path today is an external talosctl shutdown --force orchestrator, which defeats the purpose of the secondary extension.

Either of these would solve it:

• tokenize SHUTDOWNCMD (the existing TODO) and teach the bind-mounted poweroff to honor a force flag (set ShutdownRequest.Force=true); or
• ship a small force-capable poweroff helper in the extension (the machined socket is already bind-mounted) that operators can point SHUTDOWNCMD at.

Happy to test a build against this hardware.