Skip to content

[Bug] Driver silently ignores an out-of-band apply failure (agent encodes it as phase "apply" = 0x00) #91

Description

@traviswu-bigstack

Description

When snapshot_apply fails on a node, the OS-phase agent reports it out-of-band with writeSEL("apply", "error", err) (cmd/phone-home-agent/main.go:634). But "apply" is not in the agent's phaseCode map, so it silently encodes to byte 0x00. The driver's selPhaseName has no 0x00 entry → Phase=""selState returns "" → the record is ignored. The deploy UI stays stuck on "applying" instead of showing the failure. In-band reportApplyFailed only covers it while mgmt is reachable — which post-apply air-gap usually isn't, and OOB is exactly the fallback for that case.

Observed live on qa41-r630 (2026-09-01): apply failed, the node's BMC SEL held 00 ff …"snap" (phase 0x00, result 0xff=error), and the driver showed apply active indefinitely.

How to Reproduce

  1. Deploy a node where hex_config snapshot_apply returns a hard error.
  2. The node writes the apply-error SEL record; the in-band report doesn't reach the driver.
  3. Driver UI sits on "applying"; the node never shows errored.

Expected behavior

An out-of-band apply failure surfaces as an errored node.

Additional Context

Root cause is a bad phase string in the agent; the driver's OOB decode is also not defensive about the error result. Fix both: the agent emits a real error phase, and the driver treats any cube record whose result is error as terminal regardless of the phase byte (only the apply-failure record carries the error result, so it can't misfire).

Output artifacts (Definition of Done)

  • Handbook knowledge update — fold into the existing OOB-SEL known-issue or add a short note.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions