arm64 CHR panics under HVF on all Apple Silicon — CHR image requires AArch32 userspace that Apple CPUs do not implement

[mobileskyfi]

Summary

aarch64 CHR under HVF with -cpu host panics during early guest-kernel boot on Apple M4 hosts, while the same image boots under TCG. quickchr emits the byte-for-byte same QEMU config that reproduces this, so it is a latent bug here — invisible only because both CI and the maintainer's dev box are Intel x86_64 (aarch64-HVF only ever runs on Apple Silicon, and nobody has run quickchr on an M4).

Surfaced downstream in tikoci/mikropkl#11 (their qemu.sh is Pkl-generated but produces the same -M virt + -cpu host + virtio-blk-pci invocation quickchr does). Now confirmed on a real M4 — see the diagnostic results below.

Edit history: an early revision led with SME as the cause. Deeper research put FEAT_SSBS removal as the lead; the M4 reporter's data (mikropkl#11) has since confirmed it. SME is a secondary, already-fixed QEMU-side issue and is not this failure.

Reporter's signature (mikropkl#11, Apple M4, macOS, QEMU 11.0.2 Homebrew)

EFI stub: Exiting boot services and installing virtual address map...
Kernel panic - not syncing: No working init found.   (t=0.076s)
CPU features: 0x20012,28000230
Rebooting in 5 seconds..

Not the wrong-arch image, not the TPM noise, not the cortex-a710/SVE2 QEMU-CPU-init crash. It is a guest-kernel early-boot failure. QEMU_ACCEL=tcg boots the same image fine. The panic at t≈0.076s is far too early for disk probing (virtio-blk is fine under TCG) — the kernel is exec'ing init from initramfs and the transition into userspace faults, consistent with the kernel manipulating PSTATE.SSBS against a CPU that does not implement it.

Root cause — CONFIRMED on M4

The variable is not the accelerator but the CPU feature set -cpu host passes through under HVF on M4. M4 removes FEAT_SSBS (an ARMv8.5 spectre-v4 control that RouterOS's Linux 5.6.3 kernel assumes present). Confirmed by the reporter's diagnostics:

Probe	Result on M4	Conclusion
sysctl hw.optional.arm.FEAT_SSBS	0 (and FEAT_SME=1, SME2=1, MTE=0, PAuth=1)	SSBS-absence is real, not inferred.
-accel hvf -cpu max	identical panic (did not error)	The -cpu model is inert under HVF — the guest reads the physical ID registers regardless. Cannot mask SSBS-absence with any CPU model.
-cpu host,ssbs=on / ssbs=off	Property 'host-arm-cpu.ssbs' not found	QEMU 11.0.2's host model has no SSBS knob — the upstream shim is not in this build; SSBS cannot be injected.

Under TCG quickchr uses -cpu cortex-a710 (a fixed ARMv9.0 model with the standard feature set, incl. SSBS, no SME) — so the old kernel sees what it expects and boots.

Standard server ARM (Ampere Altra/AmpereOne, Graviton) implements SSBS normally and does not expose SME to unaware kernels — RouterOS aarch64 CHR runs there under KVM. This is an Apple-M4-under-HVF quirk, not a RouterOS-on-ARM defect.

No QEMU release fixes this today

The upstream M4 SSBS work is still an unmerged RFC v2 (Oct 2025), it targets older macOS guests (not Linux), and it relies on a private Apple API (_hv_vcpu_config_set_fgt_enabled, FGT trapping of SCTLR_EL1) — the kind of thing that stalls in review. It is not in 11.0.2, the 11.1 rc, or (almost certainly) --HEAD. So there is currently no QEMU version to gate on: not a released fix, not a version floor. The durable fixes are (a) QEMU merging the shim, or (b) RouterOS shipping a newer kernel that handles SSBS-absence gracefully — neither exists in a release today. The only thing that boots now is TCG.

Refs: qemu-arm SSBS RFC, Oct 2025, qemu#2665 (SME, fixed), qemu#1990 (M3 HVF assert), utmapp/UTM#6944.

Grounding caveat

Not reproduced locally — quickchr's dev/CI hosts are Intel x86_64; aarch64-HVF needs Apple Silicon and this failure needs an M4. Root cause is now grounded in the M4 reporter's D0/D1/D3 results above, plus quickchr's own codegen (src/lib/qemu.ts:73-74). Confirmed OK: M1/M2 (quickchr on M2, qemu.sh on M1/M2, both have SSBS). M3 untested but theory predicts fine (M3 keeps SSBS). Decisive M4 data is now in hand for D0/D1/D3; D2 (modern-kernel control) remains open in mikropkl#11.

Chosen fix — proactive FEAT_SSBS=0 probe → TCG (self-healing)

Detection keys on the actual root-cause axis, not a CPU marketing name:

• On macOS/arm64, probe sysctl -n hw.optional.arm.FEAT_SSBS. If it returns 0 and the requested/auto accel for aarch64 is HVF, fall back to TCG with a clear one-line warning (and honor an explicit QEMU_ACCEL=hvf / flag override for users who want to force it).
• Keep the existing boot respawn-once on hardware accel machinery (DESIGN.md Potential fixes for 4 code quality findings #8, QEMU watch-items (boot respawn-once, arm64 QGA) #45) as a general safety net for unknown HVF boot failures.
• TODO: once QEMU ships the SSBS shim in a release, gate the probe behind a QEMU-version check so HVF auto-restores. Until then there is no version to key on.

Why the probe over PR#12's machdep.cpu.brand_string == "Apple M4"* gate: the brand-string axis hard-codes a transient QEMU-era bug as permanent policy — it keeps forcing slow TCG on M4 even after a fixed QEMU lands, silently mis-handles M5+, and disables HVF for all aarch64 guests on M4 including a future RouterOS with a newer kernel. FEAT_SSBS=0 is the real condition, catches any future SSBS-less Apple chip, and is instant/deterministic (no wasted boot attempt).

Open verification (need a real M4 — tracked in mikropkl#11)

• M4 reports hw.optional.arm.FEAT_SSBS = 0 and FEAT_SME = 1. (confirmed)
• -cpu model is inert under HVF (-cpu max panics identically). (confirmed)
• SSBS not injectable in 11.0.2 (host,ssbs=on/off → property not found). (confirmed)
• D2 (open, high value): modern arm64 Linux under identical -accel hvf -cpu host — boots ⇒ RouterOS-5.6.3-specific (old kernel lacks graceful SSBS-absence handling); also panics ⇒ general M4+HVF+QEMU-11.0.2 layer.
• D4 (open, low expectation): brew install --HEAD qemu — expected to also panic, since the shim is an unmerged RFC. A boot would be a pleasant surprise ⇒ version floor.
• Verify M3 is unaffected if an M3 host becomes available.

Related: #45 (QEMU watch-items: boot respawn-once), DESIGN.md #8. Downstream: tikoci/mikropkl#11, mikropkl PR#12 (brand-string gate — being reworked to the FEAT_SSBS=0 axis).

[mobileskyfi]

Reopening — the shipped fix keyed on the wrong axis

The FEAT_SSBS=0 → TCG fallback merged in #98 is wrong in scope and wrong in mechanism, and it is still unreleased (last tag v0.4.5), so it can be corrected in place rather than deprecated.

SSBS was a coincident marker, not the cause

M4 is the first Apple chip to report hw.optional.arm.FEAT_SSBS = 0, which is why the reporter's diagnostics correlated. But Linux 5.6 treats SSBS as an optional mitigation and boots fine without it — cortex-a53 and neoverse-n1 don't implement it either. SSBS-absence cannot produce No working init found.

Actual root cause: the CHR image needs AArch32, Apple Silicon has none

Current arm64 CHR images pair an AArch64 kernel with a 32-bit ARM userspace:

CHR image	/init type
7.20.8 arm64	ELF 32-bit LSB ARM, EABI5, static
7.22.1 arm64	ELF 32-bit LSB ARM, EABI5, static
7.23beta5 arm64	ELF 32-bit LSB ARM, EABI5, static

The 7.22.1 system package holds 101 more ARM32 executables and 18 ARM32 shared objects; the only AArch64 executables are kexec and vmcore-dmesg. So an AArch64 /init alone would not fix it — the failure would just move later.

Apple Silicon implements no AArch32 at any exception level (ID_AA64PFR0_EL1 is AArch64-only for EL0/EL1), and QEMU under HVF passes that hardware register straight through — hvf_arch_init_vcpu() re-reads the live vCPU register and edits only the GIC bit, so the -cpu model is inert. The guest kernel therefore never sets ARM64_HAS_32BIT_EL0, compat_elf_check_arch() rejects the EM_ARM /init with -ENOEXEC, the initramfs has no fallback init, and Linux panics at t≈0.076 s.

Confirmed by the guest's own panic-time capability bitmap — decoded against Linux 5.6 cpucaps.h (ARM64_HAS_32BIT_EL0 == 13):

Guest	Bitmap	Caps	ARM64_HAS_32BIT_EL0
M4, -accel hvf -cpu host — panics	0x20012,28000230	8	absent
-accel tcg -cpu cortex-a710 — boots	0x20013,28402230	11	present

The failing set is a strict subset; the only other differences are ARM64_SVE and ARM64_HAS_STAGE2_FWB, neither of which participates in execve().

Consequences for the fix

1. Scope is every Apple Silicon generation, not M4+. No Apple CPU since 2020 implements AArch32, so the SSBS predicate left M1/M2/M3 on HVF and panicking — that was a live bug, not merely a narrow gate.
2. A QEMU version floor is not a valid restore signal. Apple's Hypervisor.framework exposes only hv_vcpu_config_get_feature_reg() — there is no setter — so no macOS VMM can present a feature the silicon lacks (UTM's fork behaves identically). The deferred getQemuVersion() guard is dropped, not postponed.
3. The restore signal is the guest artifact: a future arm64 CHR whose appended /init and required system-package executables/shared objects are all AArch64, confirmed by a real HVF boot. That check is mechanical and could become a release-time guard.
4. Standard server ARM (Ampere, Graviton) does implement AArch32 EL0, which is why arm64 CHR runs there under KVM. This is an Apple-Silicon-plus-CHR-artifact interaction, not a RouterOS-on-ARM defect.

Full chain, evidence table, and reproduction: docs/m4-hvf-arm64-investigation.md. Reported upstream to MikroTik (request: ship an ARM32-free arm64 userspace, or document that the image requires AArch32 EL0 despite its AArch64 kernel).

What is landing

• detectAccel("arm64") returns tcg on all macOS hosts; hostLacksSsbs()/ssbsTcgWarning() are gone, replaced by isAppleSiliconHost()/accelNote().
• New escape hatch — --accel <auto|tcg|hvf|kvm> on start/add, plus an accel setting and QUICKCHR_ACCEL env var (precedence: flag > env > quickchr.env > auto). Anything but auto is passed to QEMU verbatim and bypasses detection entirely, so HVF can be tested against a future AArch64-only image with no code change. Forcing --accel hvf for an arm64 guest on Apple Silicon still prints the panic caveat.
• doctor marks the acceleration row when an override is in effect.
• Docs corrected: DESIGN.md fix(ci): kill orphaned QEMU processes on macOS-x86 step timeout to prevent runner death #10, CHANGELOG.md, MANUAL.md, .github/instructions/qemu.instructions.md, .github/copilot-instructions.md.

Grounding boundary, unchanged: still not reproduced locally — dev/CI hosts are Intel x86_64 and this needs Apple Silicon. The verified-here parts are the accelerator selection, the override, and a real x86 CHR boot under both auto (-accel hvf) and forced (-accel tcg,tb-size=256). The M4 evidence is the reporter's (tikoci/mikropkl#11); the exact Failed to execute /init (error -8) line remains unobserved.