Description
A zero-touch deploy can stall with the node parked at preflight passed — waiting for the restore gate on the BMC, even though the node has actually finished restoring and reported restore-done. The driver never advances it, so the operator can't progress (manual) / the run never completes (auto) and eventually cancels.
Root cause: the out-of-band SEL observer (pollSEL) discards any SEL record whose timestamp predates the deploy's UTC StartedAt − 5min. That timestamp comes from the node BMC's RTC, which is routinely hours off UTC (unsynced iDRAC / localtime RTC / dead coin cell). On a skewed BMC every genuine record — restore-done, applied, done — reads as "stale from a previous deploy" and is dropped. The OOB path is exactly the fallback for when the in-band HTTP report is lost (mgmt leaves the flat L2 after apply), so both advance paths can be down at once.
Observed live on qa41-r630 (iDRAC 10.32.10.41): BMC SEL clock ran exactly 5h behind UTC (sel time get → 03:24 vs real 08:24), so the 16 01 restore-done record was silently filtered.
Secondary: the in-band restore-done handler only advanced a node already at StateRestoring; a node that reached restore via the BMC gate (no second greenlight HTTP) sits at PreflightOK, so its report was dropped too.
How to Reproduce
- Deploy a node whose BMC RTC is not synced to UTC (off by more than ~5 min).
- Node preflights, gets the restore gate, restores, reports
restore-done.
- Driver keeps the node at "waiting for the restore gate"; deploy stalls.
Expected behavior
The driver advances the node past restore (and through reboot/apply/done) regardless of the BMC's RTC offset.
Additional Context
Fix: floor OOB freshness on the SEL's own record handles, not the clock — capture each BMC's last record ID right after the deploy-start ClearSEL (NodeDeploy.SELAnchor) and ignore everything up to it in log order (immune to RTC skew, inherently per-node). Observe also now returns the latest record that maps to a real state, so a trailing gate record can't mask a status. Plus let RestoreDone advance from PreflightOK.
Validated end-to-end on qa41-r630 (a 5h-skewed BMC): restore-done → reboot → apply all advanced via the record-ID anchor.
Output artifacts (Definition of Done)
Description
A zero-touch deploy can stall with the node parked at
preflight passed — waiting for the restore gate on the BMC, even though the node has actually finished restoring and reportedrestore-done. The driver never advances it, so the operator can't progress (manual) / the run never completes (auto) and eventually cancels.Root cause: the out-of-band SEL observer (
pollSEL) discards any SEL record whose timestamp predates the deploy's UTCStartedAt − 5min. That timestamp comes from the node BMC's RTC, which is routinely hours off UTC (unsynced iDRAC / localtime RTC / dead coin cell). On a skewed BMC every genuine record —restore-done,applied,done— reads as "stale from a previous deploy" and is dropped. The OOB path is exactly the fallback for when the in-band HTTP report is lost (mgmt leaves the flat L2 after apply), so both advance paths can be down at once.Observed live on qa41-r630 (iDRAC
10.32.10.41): BMC SEL clock ran exactly 5h behind UTC (sel time get→03:24vs real08:24), so the16 01restore-done record was silently filtered.Secondary: the in-band
restore-donehandler only advanced a node already atStateRestoring; a node that reached restore via the BMC gate (no second greenlight HTTP) sits atPreflightOK, so its report was dropped too.How to Reproduce
restore-done.Expected behavior
The driver advances the node past restore (and through reboot/apply/done) regardless of the BMC's RTC offset.
Additional Context
Fix: floor OOB freshness on the SEL's own record handles, not the clock — capture each BMC's last record ID right after the deploy-start
ClearSEL(NodeDeploy.SELAnchor) and ignore everything up to it in log order (immune to RTC skew, inherently per-node).Observealso now returns the latest record that maps to a real state, so a trailing gate record can't mask a status. Plus letRestoreDoneadvance fromPreflightOK.Validated end-to-end on qa41-r630 (a 5h-skewed BMC): restore-done → reboot → apply all advanced via the record-ID anchor.
Output artifacts (Definition of Done)