Fault-inject real driver PDs, not just the crasher

[schonfinkel]

Summary

Exercise the whole restart path against a REAL driver PD: the driver takes a genuine seL4 fault, the Root handler catches it, the driver re-initialises, and its clients recover. Today the two halves are tested separately and never composed.

Motivation

Issues #16 and #18 each got the half of the path they were about, and the split was deliberate: driving recovery through a real fault would have conflated "did we detect it" with "did the driver come back". What the split leaves untested is the join.

• Detection is covered by restart-smoke, against the crasher PD. The crasher has no clients, no hardware and no sDDF queues, so it proves fault() runs and the budget converges, and nothing about a driver.
• Recovery is covered by serial-restart-smoke, timer-restart-smoke, blk-restart-smoke and net-restart-smoke. Each writes a class name to /dev/pd-restart, the shim notifies root on a debug channel, and root calls microkit_pd_restart on a HEALTHY driver.

So no test has ever observed a driver PD fault. That matters because the two entries to root_restart_child() do not leave the child in the same state. A debug restart preempts a driver that is idle in seL4_Recv with its data structures consistent; a fault arrives with the driver stopped mid-operation, holding a half-written descriptor pair, an unacked IRQ, or an ialloc slot allocated for a request whose descriptor was never published. Every restartable-init patch we wrote reasons explicitly about that state (the torn header/packet pair in the net reclaim, the unacked IRQ in serial and timer, the request-ring drain in blk), and none of it is reached by the tests that pass today.

Issue #18 asks for exactly this in its fourth acceptance criterion, "fault-injection restart tests pass for each driver class". Splitting it out so that issue can close on what it actually delivers.

Proposed Solution

Fault a driver by restarting it to a bogus entry point. microkit_pd_restart(child, 0) rewrites the child's PC to 0 and resumes it, so the driver immediately takes an instruction fault, from the real driver PD, with its real clients attached and its memory in whatever state the preemption left it. Root's fault() catches it exactly as it would a driver bug, and restarts it at the true restart_entry.

The appeal is that it needs no new mechanism in the drivers. An alternative would be a test-only "fault now" notification patched into each sDDF driver, which means four more patches carrying deliberate NULL derefs, gated on the restart image, in code we want to send upstream.

• Add a second debug channel per class, or a fault:<class> payload on the existing /dev/pd-restart write, meaning "restart this child to entry 0". Keep it behind --with-restart-debug like the existing channels, so production-sdf-gate covers it for free.
• In src/runtime/root.c, take the injected fault down the ordinary fault() path. It already logs ROOT|fault|child=N|label=...|mr0=..., which is what the test keys on: the assertion is that a ROOT|fault for the driver's pinned child id appears, followed by ROOT|restart, followed by the class's recovery assertion.
• One test per class, reusing the existing recovery assertions verbatim (console echo, monotonic clock, fs read, TCP round trip), so the only difference from the current tests is how the driver went down. Faulting costs budget the same way a debug restart does, since both go through root_restart_child().
• Fault at least one class with a request in flight, not only at idle. Blk is the one that matters: blk-restart-smoke already has the in-flight scenario, and running it through a fault is what actually exercises the virtualiser's reconcile against a driver that died holding the request rather than one that was asked to restart.

Alternatives Considered

• Leave it as is. The composed path is the one that runs in production; a driver is restarted by faulting, never by a notification, since /dev/pd-restart does not exist in the shipped image. Testing only the halves means the shipped path is the untested one.
• A test-only fault trigger inside each driver. Rejected above: four extra sDDF patches whose whole content is a deliberate crash, and they would fault at a point the driver author chose rather than an arbitrary one.
• Fault the driver by corrupting its config from another PD. No PD holds a writable mapping of another's config, and adding one to inject a fault would weaken the isolation the error kernel exists to provide.

Non-goals

• Torn writes. A blk_driver faulting mid-write can leave a partially applied write on disk, the virtualiser fails the request back with BLK_RESP_ERR_UNSPEC and the client retries, which is correct for the request but says nothing about what landed on the medium. Worth its own issue if we ever need write atomicity.
• Faulting the virtualisers, fs_server or beam_server. They are not children of root and their faults are still terminal (Make beam_server cleanly restartable #17).

Acceptance Criteria

• A driver PD can be made to take a genuine seL4 fault on demand, in the restart image only, with production-sdf-gate still green.
• For each of serial, timer, blk and eth: the injected fault produces a ROOT|fault for that child id, root restarts it, and the class's existing recovery assertion passes afterwards.
• At least one class is faulted with a client request in flight, and the client gets an answer (success or error) rather than hanging.
• Faults consume the same restart budget as debug restarts, asserted by driving a class to ROOT|giveup|reason=budget-exhausted through faults alone.
• boot-smoke + shell-smoke green.