ERTS timers stop firing after a timer_driver restart

[schonfinkel]

Summary

• TL;DR: timer:sleep/1 never returns once timer_driver has been restarted, though every sDDF layer beneath it is verified healthy

After timer_driver is restarted by the Root fault handler, ERTS-level timers stop firing in beam_server: timer:sleep(500) never returns, even after minutes. The same call succeeds immediately before the restart on the same boot.

Everything below ERTS is measured and healthy across the restart, so this is not a driver or sDDF-transport bug. It is a client-side (ERTS / libmicrokitco) problem, split out from #18 so that issue can close on the criterion it actually states ("restarting timer_driver recovers the monotonic clock", which is met and asserted by timer-restart-smoke).

Evidence

Measured with the diagnostics that are left commented out in nix/patches/sddf-timer-arm-restartable-init.patch and src/runtime/main.c (re-enable to reproduce). Counts are before vs. after one restart in a single timer-restart-smoke run:

Layer	Result
Driver re-initialises	TIMER|init|enter + TIMER|init|done|freq=62500000 after every restart
Timer hardware IRQ	fires 1298 before -> 1819 after, more after, not fewer
PPC path	GET_TIME and SET_TIMEOUT both served continuously after restart
Client re-arms	~1825 SET_TIMEOUT calls after the restart (~10/sec, the lwIP tick)
beam_server receives notifications	1298 before / 1820 after, matches the driver's IRQ count 1:1
Monotonic clock	advances correctly across the restart (asserted in the test)
timer:sleep(500)	never returns (works before the restart: SLEPT_BEFORE|603)

So the full chain, hardware IRQ -> process_timeouts -> sddf_notify -> beam_server notified() -> thread_io_wake() -> parked cothread, is intact and running at its normal ~10 Hz. Not one notification is lost. The break is above that, in how ERTS consumes the wakeups.

Note the failure is deterministic, not a race, which argues against any "restarted mid-operation and lost a one-off signal" explanation.

Ruled out

This is saved here so this is not re-investigated from scratch:

• The lost-timeout notification. The driver now signals clients whose timeouts it discarded (it identifies them from timeouts[] surviving in .bss, index == channel id, guarded by an initialised flag so a cold boot does not signal unwired channels). This is a real fix and is in place, it moved the failure from "beam_server wedges completely" to the current narrower symptom. It is not sufficient.
• The IRQ ack in init(). A driver dying mid-interrupt really can lose its IRQ permanently, and the patch is worth keeping, but the IRQ demonstrably fires after a restart, so this was never this bug.
• A broken endpoint / reply path. PPCs are delivered and served after the restart.
• The client-side stale-deadline heuristic once tried in beam_timer_arm.
  • Bisected: removing it changed nothing. It was also actively harmful, "deadline passed but slot still armed" is also the ordinary window between a timeout firing and notified() running, so clearing there let a later deadline overwrite an imminent one. Reverted.

Where to look next

• src/runtime/process.c: thread_io_wait() / thread_io_wake() and the io_wake_generation handshake, confirm the ERTS scheduler cothread actually parks and is pulsed, rather than being blocked somewhere else (e.g. park_sem).
• src/runtime/bringup.c: bringup_epoll_pwait / bringup_ppoll / bringup_pselect6: the stubs ERTS sleeps in. They park once and return, expecting ERTS to re-evaluate.
• ERTS time-correction machinery: whether it latches state that a restart perturbs, given erlang:monotonic_time() itself reads correctly.

Acceptance Criteria

• timer:sleep/1 returns normally after timer_driver is restarted.
• timer-restart-smoke re-enables the post-restart sleep_works(...) assertion (it is present but commented out, with the evidence above, in tests.nix).
• Root cause documented, if it turns out to be an ERTS limitation rather than a bug in this port, say so explicitly and state the supported behaviour instead.