hypervisor: increase ipa limit from 36 bits to 39

[exverge-0]

Increases IPA limit from the default 36 bits to 39.

I also replaced posix_memalign with mmap as the Apple documentation advises against it.

[SamoZ256]

Just a note: according to switchbrew, the virtual user address space can have 38 bits at most (https://switchbrew.org/wiki/Memory_layout), so I guess 39 bits for the physical one should be enough. I will just test the PR and then merge it afterwards

[exverge-0]

Just a note: according to switchbrew, the virtual user address space can have 38 bits at most (https://switchbrew.org/wiki/Memory_layout)

I'm not 100% sure how hydra handles the address space but from what I can tell, Hydra maps the host memory address to the guest's "physical" address (ipa) 1:1, and then the guest virtual address is mapped with TTBR0, (which is hydra's MMU/PageTable under the hood). The 38-bit limit wouldn't apply here as the issue comes with Apple Hypervisor by default using 36 bits for the IPA address space and only supporting up to 39/40 bits compared to macOS's userspace limit of 47.

One way we could probably fix this completely is to skip the TTBR0_EL0 step (if that's possible?), and have the MMU map the addresses directly with hv_vm_map instead of IMemory though from when I've tested it mmap usually won't return an address greater than 39 bits anyway.

[SamoZ256]

Just a note: according to switchbrew, the virtual user address space can have 38 bits at most (https://switchbrew.org/wiki/Memory_layout)

I'm not 100% sure how hydra handles the address space but from what I can tell, Hydra maps the host memory address to the guest's "physical" address (ipa) 1:1, and then the guest virtual address is mapped with TTBR0, (which is hydra's MMU/PageTable under the hood). The 38-bit limit wouldn't apply here as the issue comes with Apple Hypervisor by default using 36 bits for the IPA address space and only supporting up to 39/40 bits compared to macOS's userspace limit of 47.

One way we could probably fix this completely is to skip the TTBR0_EL0 step (if that's possible?), and have the MMU map the addresses directly with hv_vm_map instead of IMemory though from when I've tested it mmap usually won't return an address greater than 39 bits anyway.

Disabling the MMU would be viable only on Tahoe and later as before that the physical address space supported only 16KiB pages (and the Switch uses 4KiB ones)

[exverge-0]

Marking this as draft for now because as far as I can tell, set_max_ipa_size doesn't actually change any behavior and instead just silently fails?? It stops the error from occurring at hv_vm_map but from all other points it acts as if it weren't mapped. Will do more testing and see

Edit: TCR_EL1 set the max IPA address size to 36 bits, updating the value fixes this issue.

[SamoZ256]

@exverge-0 I get hv_vm_config_set_ipa_size(config, 39) failed: 0xfffffffffae9400f when I try to run anything

[exverge-0]

@exverge-0 I get hv_vm_config_set_ipa_size(config, 39) failed: 0xfffffffffae9400f when I try to run anything

That's HV_UNSUPPORTED, which is weird as I remember seeing somewhere that 39 bits was supported by most modern systems.. Can you try now?

[SamoZ256]

It works now, but I tried logging max_bits and it's just 36 on my machine (M1 Air). Also, doesn't it just use the maximum number of bits by default?

[exverge-0]

According to an "Apple" QEMU patch (this is the closest thing to documentation I could find), 36 bits is the default on all systems with 39 bits being the max on macOS 13+ and 40 on 15+, so I have no idea why your machine would only support 36. (unless you're on a version lower than that?)

Either way I'd still recommend you merge this as at least on my machine (M4 on both 26 & 27), AllocateVmMemory (both the mmap and posix_memalign versions) consistently allocate large spaces like the heap higher than 36 bits making the hypervisor effectively unusable

[SamoZ256]

According to an "Apple" QEMU patch (this is the closest thing to documentation I could find), 36 bits is the default on all systems with 39 bits being the max on macOS 13+ and 40 on 15+, so I have no idea why your machine would only support 36. (unless you're on a version lower than that?)

Either way I'd still recommend you merge this as at least on my machine (M4 on both 26 & 27), AllocateVmMemory (both the mmap and posix_memalign versions) consistently allocate large spaces like the heap higher than 36 bits making the hypervisor effectively unusable

Ok great, I will merge this then once you've reverted those kernel changes 👍