Proxmox QEMU patch

[mhdo298]

Hi, this is not really an issue but more of a feature request/tutorial for how to use this to patch Proxmox's version of QEMU. Also a tidbit at the end for how to bypass most detections in VMAware (only 4 tests remains in the 2.4.1 release - it can still tell you're in a QEMU VM). Someone should probably add this as an option (I will if I have time but summer is ending), as it is basically just using the QEMU option but with a different source and slightly different build steps.

Using this repository for Proxmox

1. To get the latest QEMU with Proxmox patches, do:

git clone --recursive git://git.proxmox.com/git/pve-qemu.git

Alternatively, one can just do a non-recursive clone, then download the qemu source and move it to pve-qemu/qemu. (Have not tested this, pretty sure the next step does not really work if the qemu source is not treated a git repo).
2. Download the appropriate patch file for your CPU_VENDOR and QEMU_VERSION found in this repository to pve-qemu/qemu. I have only tested this for Intel and version 10.0.2.
3. Copy the following script (basically verbatim from the QEMU patch but with parts removed) and execute it inside pve-qemu/qemu:

Click for script

#!/bin/bash
# Modified from https://github.com/Scrut1ny/Hypervisor-Phantom/blob/main/Hypervisor-Phantom/modules/spoof_qemu_patch.sh
readonly QEMU_VERSION="10.0.2"
VENDOR_ID=$(LANG=en_US.UTF-8 lscpu 2>/dev/null | awk -F': +' '/^Vendor ID:/ {print $2}' | xargs)

declare -r CPU_VENDOR=$(case "$VENDOR_ID" in
  *AuthenticAMD*) echo "amd" ;;
  *GenuineIntel*) echo "intel" ;;
  *) fmtr::error "Unknown CPU Vendor ID."; exit 1 ;;
esac)

patch_qemu() {
  git apply $CPU_VENDOR-qemu-$QEMU_VERSION.patch
}

spoof_serial_numbers() {
  local patterns=(STRING_SERIALNUMBER STR_SERIALNUMBER STR_SERIAL_MOUSE \
                  STR_SERIAL_TABLET STR_SERIAL_KEYBOARD STR_SERIAL_COMPAT)
  for file in ./hw/usb/*.c; do
    for pat in "${patterns[@]}"; do
      grep -n "\[\s*${pat}\s*\]\s*=\s*\"[^\"]*\"" "$file" | cut -d: -f1 | while read -r lineno; do
        serial=$(tr -dc 'A-Z0-9' </dev/urandom | head -c10)
        sed -r -i "${lineno}s/(\[\s*${pat}\s*\]\s*=\s*\")[^\"]*(\")/\1${serial}\2/" "$file"
      done
    done
  done
}

spoof_drive_serial_number() {

  local ide="hw/ide/core.c"
  local nvme="hw/nvme/ctrl.c"

  local ide_cd_models=(
    "HL-DT-ST BD-RE WH16NS60" "HL-DT-ST DVDRAM GH24NSC0"
    "HL-DT-ST BD-RE BH16NS40" "HL-DT-ST DVD+-RW GT80N"
    "HL-DT-ST DVD-RAM GH22NS30" "HL-DT-ST DVD+RW GCA-4040N"
    "Pioneer BDR-XD07B" "Pioneer DVR-221LBK" "Pioneer BDR-209DBK"
    "Pioneer DVR-S21WBK" "Pioneer BDR-XD05B" "ASUS BW-16D1HT"
    "ASUS DRW-24B1ST" "ASUS SDRW-08D2S-U" "ASUS BC-12D2HT"
    "ASUS SBW-06D2X-U" "Samsung SH-224FB" "Samsung SE-506BB"
    "Samsung SH-B123L" "Samsung SE-208GB" "Samsung SN-208DB"
    "Sony NEC Optiarc AD-5280S" "Sony DRU-870S" "Sony BWU-500S"
    "Sony NEC Optiarc AD-7261S" "Sony AD-7200S" "Lite-On iHAS124-14"
    "Lite-On iHBS112-04" "Lite-On eTAU108" "Lite-On iHAS324-17"
    "Lite-On eBAU108" "HP DVD1260i" "HP DVD640"
    "HP BD-RE BH30L" "HP DVD Writer 300n" "HP DVD Writer 1265i"
  )

  local ide_cfata_models=(
    "SanDisk Ultra microSDXC UHS-I" "SanDisk Extreme microSDXC UHS-I"
    "SanDisk High Endurance microSDXC" "SanDisk Industrial microSD"
    "SanDisk Mobile Ultra microSDHC" "Samsung EVO Select microSDXC"
    "Samsung PRO Endurance microSDHC" "Samsung PRO Plus microSDXC"
    "Samsung EVO Plus microSDXC" "Samsung PRO Ultimate microSDHC"
    "Kingston Canvas React Plus microSD" "Kingston Canvas Go! Plus microSD"
    "Kingston Canvas Select Plus microSD" "Kingston Industrial microSD"
    "Kingston Endurance microSD" "Lexar Professional 1066x microSDXC"
    "Lexar High-Performance 633x microSDHC" "Lexar PLAY microSDXC"
    "Lexar Endurance microSD" "Lexar Professional 1000x microSDHC"
    "PNY Elite-X microSD" "PNY PRO Elite microSD"
    "PNY High Performance microSD" "PNY Turbo Performance microSD"
    "PNY Premier-X microSD" "Transcend High Endurance microSDXC"
    "Transcend Ultimate microSDXC" "Transcend Industrial Temp microSD"
    "Transcend Premium microSDHC" "Transcend Superior microSD"
    "ADATA Premier Pro microSDXC" "ADATA XPG microSDXC"
    "ADATA High Endurance microSDXC" "ADATA Premier microSDHC"
    "ADATA Industrial microSD" "Toshiba Exceria Pro microSDXC"
    "Toshiba Exceria microSDHC" "Toshiba M203 microSD"
    "Toshiba N203 microSD" "Toshiba High Endurance microSD"
  )

  local default_models=(
    "Samsung SSD 970 EVO 1TB" "Samsung SSD 860 QVO 1TB"
    "Samsung SSD 850 PRO 1TB" "Samsung SSD T7 Touch 1TB"
    "Samsung SSD 840 EVO 1TB" "WD Blue SN570 NVMe SSD 1TB"
    "WD Black SN850 NVMe SSD 1TB" "WD Green 1TB SSD"
    "WD Blue 3D NAND 1TB SSD" "Crucial P3 1TB PCIe 3.0 3D NAND NVMe SSD"
    "Seagate BarraCuda SSD 1TB" "Seagate FireCuda 520 SSD 1TB"
    "Seagate IronWolf 110 SSD 1TB" "SanDisk Ultra 3D NAND SSD 1TB"
    "Seagate Fast SSD 1TB" "Crucial MX500 1TB 3D NAND SSD"
    "Crucial P5 Plus NVMe SSD 1TB" "Crucial BX500 1TB 3D NAND SSD"
    "Crucial P3 1TB PCIe 3.0 3D NAND NVMe SSD"
    "Kingston A2000 NVMe SSD 1TB" "Kingston KC2500 NVMe SSD 1TB"
    "Kingston A400 SSD 1TB" "Kingston HyperX Savage SSD 1TB"
    "SanDisk SSD PLUS 1TB" "SanDisk Ultra 3D 1TB NAND SSD"
  )

  get_random_element() {
    local array=("$@")
    echo "${array[RANDOM % ${#array[@]}]}"
  }

  local new_ide_cd_model=$(get_random_element "${ide_cd_models[@]}")
  local new_ide_cfata_model=$(get_random_element "${ide_cfata_models[@]}")
  local new_default_model=$(get_random_element "${default_models[@]}")

  sed -i "$ide" -Ee "s/\"HL-DT-ST BD-RE WH16NS60\"/\"${new_ide_cd_model}\"/"
  sed -i "$ide" -Ee "s/\"Hitachi HMS360404D5CF00\"/\"${new_ide_cfata_model}\"/"
  sed -i "$ide" -Ee "s/\"Samsung SSD 980 500GB\"/\"${new_default_model}\"/"
  sed -i "$nvme" -Ee "s/\"NVMe Ctrl\"/\"${new_default_model}\"/"

}

spoof_acpi_table_data() {

  ##################################################
  ##################################################

  # Spoofs 'OEM ID' and 'OEM Table ID' for ACPI tables.

  local oem_pairs=(
    'DELL  ' 'Dell Inc' ' ASUS ' 'Notebook'
    'MSI NB' 'MEGABOOK' 'LENOVO' 'TC-O5Z  '
    'LENOVO' 'CB-01   ' 'SECCSD' 'LH43STAR'
    'LGE   ' 'ICL     '
  )

  if [[ "$CPU_VENDOR" == "amd" ]]; then
    oem_pairs+=('ALASKA' 'A M I ')
  elif [[ "$CPU_VENDOR" == "intel" ]]; then
    oem_pairs+=('INTEL ' 'U Rvp   ')
  fi

  local total_pairs=$(( ${#oem_pairs[@]} / 2 ))
  local random_index=$(( RANDOM % total_pairs * 2 ))
  local appname6=${oem_pairs[$random_index]}
  local appname8=${oem_pairs[$random_index + 1]}
  local h_file="include/hw/acpi/aml-build.h"

  sed -i "$h_file" -e "s/^#define ACPI_BUILD_APPNAME6 \".*\"/#define ACPI_BUILD_APPNAME6 \"${appname6}\"/"
  sed -i "$h_file" -e "s/^#define ACPI_BUILD_APPNAME8 \".*\"/#define ACPI_BUILD_APPNAME8 \"${appname8}\"/"

  ##################################################
  ##################################################

  # Default QEMU has an unspecified PM type in the FACP ACPI table.
  # On baremetal normally vendors specify either 1 (Desktop) or 2 (Notebook).
  # We patch the PM type integer based on the chassis type output from dmidecode.

  # fmtr::info "Obtaining machine's chassis-type..."

  local c_file="hw/acpi/aml-build.c"
  local pm_type="1" # Desktop
  local chassis_type=$(sudo dmidecode --string chassis-type)

  if [[ "$chassis_type" = "Notebook" ]]; then
    pm_type="2" # Notebook/Laptop/Mobile
  fi

  sed -i 's/build_append_int_noprefix(tbl, 0 \/\* Unspecified \*\//build_append_int_noprefix(tbl, '"$pm_type"' \/\* '"$chassis_type"' \*\//g' "$c_file"

  if [[ "$chassis_type" = "Notebook" ]]; then
    fmtr::warn "Host PM type equals '$pm_type' ($chassis_type)"
    fmtr::info "Generating fake battery SSDT ACPI table..."

    cat "${FAKE_BATTERY_ACPITABLE}" \
      | sed "s/BOCHS/$appname6/" \
      | sed "s/BXPCSSDT/$appname8/" > "$HOME/fake_battery.dsl"
    iasl -tc "$HOME/fake_battery.dsl" &>> "$LOG_FILE"

    fmtr::info "ACPI table saved to '$HOME/fake_battery.aml'"
    fmtr::info "It's highly recommended to passthrough the ACPI Table via QEMU's args/xml:
      qemu-system-x86_64 -acpitable '$HOME/fake_battery.aml'"
  fi

  ##################################################
  ##################################################

}

spoof_smbios_processor_data() {

  ##################################################
  ##################################################

  local chipset_file
  case "$QEMU_VERSION" in
    "8.2.6") chipset_file="hw/i386/pc_q35.c" ;;
    "9.2.4"|"10.0.2") chipset_file="hw/i386/fw_cfg.c" ;;
    *) fmtr::warn "Unsupported QEMU version: $QEMU_VERSION" ;;
  esac

  local manufacturer=$(sudo dmidecode --string processor-manufacturer)
  sed -i "$chipset_file" -e "s/smbios_set_defaults(\"[^\"]*\",/smbios_set_defaults(\"${manufacturer}\",/"

  ##################################################
  ##################################################

  # Handle 0x0000, DMI type 0

  local smbios_file="hw/smbios/smbios.c"
  local t0_raw="/sys/firmware/dmi/entries/0-0/raw"

  [[ -e $t0_raw ]] || sudo modprobe dmi_sysfs >>"$LOG_FILE"

  local data=$(sudo hexdump -v -e '/1 "%02X"' "$t0_raw")

  local rom_size="${data:18:2}"
  local bios_characteristics="$(echo "${data:20:16}" | fold -w2 | tac | tr -d '\n')"
  local characteristics_ext1="${data:36:2}"
  local characteristics_ext2="${data:38:2}"

  ##################################################
  ##################################################

  # Handle 0x0004, DMI type 4

  local smbios_file="hw/smbios/smbios.c"
  local t4_raw="/sys/firmware/dmi/entries/4-0/raw"

  [[ -e $t4_raw ]] || sudo modprobe dmi_sysfs >>"$LOG_FILE"

  local data=$(sudo hexdump -v -e '/1 "%02X"' "$t4_raw")

  local processor_type="${data:10:2}"
  local processor_family="${data:12:2}"
  local voltage="${data:34:2}"
  local external_clock="${data:38:2}${data:36:2}"
  local max_speed="${data:42:2}${data:40:2}"
  local current_speed="${data:46:2}${data:44:2}"
  local status="${data:48:2}"
  local processor_upgrade="${data:50:2}"
  local l1_cache_handle="${data:54:2}${data:52:2}"
  local l2_cache_handle="${data:58:2}${data:56:2}"
  local l3_cache_handle="${data:62:2}${data:60:2}"
  local processor_characteristics="${data:78:2}${data:76:2}"
  local processor_family2="${data:82:2}${data:80:2}"

  sed -i -E "s/(t->processor_family[[:space:]]*=[[:space:]]*)0x[0-9A-Fa-f]+;/\10x${processor_family};/" "$smbios_file"
  sed -i -E "s/(t->voltage[[:space:]]*=[[:space:]]*)0;/\10x${voltage};/" "$smbios_file"
  sed -i -E "s/(t->external_clock[[:space:]]*=[[:space:]]*cpu_to_le16\()0x[0-9A-Fa-f]+(\);)/\10x${external_clock}\2/" "$smbios_file"
  sed -i -E "s/(t->l1_cache_handle[[:space:]]*=[[:space:]]*cpu_to_le16\()0x[0-9A-Fa-f]+(\);)/\10x${l1_cache_handle}\2/" "$smbios_file"
  sed -i -E "s/(t->l2_cache_handle[[:space:]]*=[[:space:]]*cpu_to_le16\()0x[0-9A-Fa-f]+(\);)/\10x${l2_cache_handle}\2/" "$smbios_file"
  sed -i -E "s/(t->l3_cache_handle[[:space:]]*=[[:space:]]*cpu_to_le16\()0x[0-9A-Fa-f]+(\);)/\10x${l3_cache_handle}\2/" "$smbios_file"
  sed -i -E "s/(t->processor_upgrade[[:space:]]*=[[:space:]]*)0x[0-9A-Fa-f]+;/\10x${processor_upgrade};/" "$smbios_file"
  sed -i -E "s/(t->processor_characteristics[[:space:]]*=[[:space:]]*cpu_to_le16\()0x[0-9A-Fa-f]+(\);)/\10x${processor_characteristics}\2/" "$smbios_file"
  sed -i -E "s/(t->processor_family2[[:space:]]*=[[:space:]]*cpu_to_le16\()0x[0-9A-Fa-f]+(\);)/\10x${processor_family2}\2/" "$smbios_file"

  ##################################################
  ##################################################
}

patch_qemu
spoof_serial_numbers
spoof_drive_serial_number
spoof_smbios_processor_data
spoof_acpi_table_data

4. Go back to pve-qemu and run make to build.
5. In pve-qemu, run dpkg -i *.deb to install.
   Note that after this, your VMs might need to have IDE as their storage device type (I'm not sure why but SCSI didn't work with cloud-init, this part requires more investigation).

Additional things to bypass more VMAware checks

• Thread count: DO NOT choose host as your CPU type, as VMAware has a dictionary of known CPU thread counts, and you will need to have the VM take up all the threads on your systems to pass this check. Instead, choose literally any other compatible CPU type, Haswell, for example, worked for a VM with 4 cores. There might be a better way to bypass this check, and you might need to choose other CPU types for more cores.
• Disk size: It is perfectly fine for total disk space to be more than physically available space on Proxmox, so just create your VM with 128GB of storage (I was running on a physical system with only a 60GB SSD and this still worked).
• Basic hypervisor checks: Modify the startup configuration for your VM by editing /etc/pve/qemu-server/[VM ID].conf, which contains all the options passed to kvm to run your VM. Simply add:
  args: -cpu [CPU TYPE],-hypervisor
• Firmware check: This requires you to add more ACPI tables, specifically the SSDT ones. One way to do this is to get the ones from your own device. Here's a quick guide on how to do so:

apt install acpica-tools
mkdir /root/acpi
cd /root/acpi
acpidump -b

The above snippet dumps all the acpi tables for your host to /root/acpi. To add these tables to your device, edit /etc/pve/qemu-server/[VM ID].conf and add: args: -cpu [CPU TYPE],-hypervisor -acpitable file=/root/acpi/ssdt1.dat -acpitable file=/root/acpi/ssdt2.dat -acpitable file=/root/acpi/ssdt3.dat ... Note that not all SSDT tables will be compatible (as in your Windows VM will crash), so you will need to test around a little bit. Higher indicies tends to be less essential components, so they should be more compatible. You need to add enough to bypass the count check (I've tested with 6 and the test passes, pretty sure you can get away with 4). It might be possible to just get dumps from a bunch of physical machines and randomize from there if this were to be added to this repository.

• Audio check: In the Hardware tab, add a Sound device. You can change its backend to dummy.
• Power capabilities: in /etc/pve/qemu-server/[VM ID].conf, edit the line starting with machine and add ,enable-s4=1 to the end. For whatever reason, enable-s3=1 does not really work (apparently the default GPU doesn't allow S3 sleep), which is important as most modern machines have at least S0 or S3 enabled (and honestly VMAware should probably change their check to look for S3 or AoAc, along with S4 and S5, instead of just one of S1-4)

This leaves 4 failed checks:

• Checking timing anomalies: allegedly can be fixed by patching the kernel (vmaware detected 5/96 (and detects that it is this hypervisor) #92), but VMAware 2/96 (not incl. false pos.) - still reporting "VM brand: QEMU" also #95 seems to say otherwise.
• Checking GPU capabilities: this doesn't seem to be a common issue, so I'm guessing there's a way around this that I have not found.
• Checking QEMU passthrough: GPU ACPI locations paths exposed #106 seems to suggest that this is not resolved.
• Checking hypervisor interception: can be bypassed by enabling Hyper-V (vmaware detected 5/96 (and detects that it is this hypervisor) #92), but I'm not sure if that would work for similar checks that might come in later patches.

[mhdo298]

I took a look at the dev branch for VMAware, and as things stand, the setup above fails 7 tests (it says 9/99 but I only counted 7)

• The new checks for ACPI requiers a much more comprehensive patch of QEMU's ACPI tables, so the above steps will not be enough.
• One of the failed check is for the display, pretty sure that one can be bypassed relatively quickly.
• Have not looked at the objects test yet, but it is also failing.

[cedws]

Thanks for this guide. I'm applying amd-qemu-10.1.1.patch to QEMU 10.1.2 (Proxmox distro) and running the script above to patch with system-specific info. Unfortunately when I try to boot my VM I just get a black screen. KVM doesn't crash, it just seems no devices are initialised and it doesn't reach the UEFI.

VM config:

agent: 1
allow-ksm: 0
audio0: device=ich9-intel-hda,driver=none
balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 24
cpu: host,flags=+amd-no-ssb;+pdpe1gb;+aes;
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,ms-cert=2023,pre-enrolled-keys=1,size=4M
hostpci0: 0000:08:00.0,pcie=1,x-vga=1
hostpci1: 0000:08:00.1,pcie=1
hostpci2: 0000:0a:00.4,pcie=1
hostpci4: 0000:0a:00.3,pcie=1
hostpci5: 0000:05:00.1,pcie=1
hostpci6: 0000:05:00.3,pcie=1
hugepages: any
kvm: 1
machine: pc-q35-10.1,enable-s4=1
memory: 24576
meta: creation-qemu=10.1.2,ctime=1765031295
name: Gaming
net0: virtio=BC:24:11:68:0E:95,bridge=vmbr0,queues=4
numa: 1
onboot: 1
ostype: win11
parent: Ready
scsi0: local-lvm:vm-100-disk-1,cache=writeback,discard=on,iothread=1,size=64G,ssd=1
scsi1: local-lvm:vm-100-disk-2,cache=writeback,discard=on,iothread=1,size=1000G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=e43e7e92-e55a-4e5d-a4fb-78522824778c
sockets: 1
vmgenid: 9c84321f-7b0b-4b53-b7da-f2f70d95481a

[mhdo298]

Note that after this, your VMs might need to have IDE as their storage device type (I'm not sure why but SCSI didn't work with cloud-init, this part requires more investigation).

From what I remembered, this was also an issue for the Windows VM I was testing out (not sure why I only mentioned cloud-init). Try changing the drive types of your VM to IDE instead of SCSI maybe? You can do that on the hardware tab of the VM I believe (or when you create a new VM from the GUI, or by changing -scsi1 to -ide1 etc. if you use the CLI instead)

[cedws]

No luck after changing to IDE. Did you have issues getting to the UEFI or just booting into the guest OS?

I tried recompiling with the original patch only (amd-qemu-10.1.1.patch) and get the same so it seems like an issue there.

[mhdo298]

I had issues getting into UEFI as well if I remember correctly, and it was caused by the disks not being recognized by the VM. I won’t have access to the systems that I ran these tests on any time soon, so I can’t exactly post a screenshot of my VM setup either.

There’s a chance that changing the SCSI controller in addition to the drive types would fix the issue. This was from some old notes I wrote testing the setup:

In the System tab, choose SCSI Controller to be Default (LSI 53C895A).
In the Disks tab, choose Bus/Device to be IDE instead of SCSI.

Otherwise the issue is prolly caused by the new patch version.

[cedws]

I created a new VM and chose the LSI 53C895A controller and set the disk to IDE just to see if I could get it to show the UEFI splash but that didn't work either unfortunately. I guess the patch needs updating.

I'll maybe try downgrading QEMU to 10.1.1 but not sure that's going to work either. It would be odd if a QEMU patch release introduced a change that broke it.

[cedws]

Downgraded QEMU to 10.1.0 because that's the nearest version that Proxmox had patches for. amd-qemu-10.1.1.patch applied cleanly but VM still doesn't boot. I think it could be that the patch doesn't work for my hardware.