docker fails with "[hcat] Illegal instruction" on debian w/ AMD #18
Description
I've recently started to have an issue where running with docker starts to fail with [hcat] Illegal instruction:
Details
root@Debian-bullseye-latest-amd64-base ~/fuzzamoto # docker run --privileged -it fuzzamoto bash
root@a6de3b37aaff:/# mkdir /tmp/in && echo "AAA" > /tmp/in/A
root@a6de3b37aaff:/# AFL_PATH=AFLplusplus afl-fuzz -X -i /tmp/in -o /tmp/out -- /tmp/fuzzamoto_scenario-http-server
[+] Enabled environment variable AFL_PATH with value AFLplusplus
afl-fuzz++4.33a based on afl by Michal Zalewski and a large online community
[+] AFL++ is maintained by Marc "van Hauser" Heuse, Dominik Maier, Andrea Fioraldi and Heiko "hexcoder" Eißfeldt
[+] AFL++ is open source, get it at https://github.com/AFLplusplus/AFLplusplus
[+] NOTE: AFL++ >= v3 has changed defaults and behaviours - see README.md
[+] AFL++ Nyx mode is enabled (developed and maintained by Sergej Schumilo)
[+] Nyx is open source, get it at https://github.com/Nyx-Fuzz
[+] No -M/-S set, autoconfiguring for "-S default"
[*] Getting to work...
[+] Using exploration-based constant power schedule (EXPLORE)
[+] Enabled testcache with 50 MB
[+] Generating fuzz data with a length of min=1 max=1048576
[*] Trying to load libnyx.so plugin...
[+] libnyx plugin is ready!
[+] You have 32 CPU cores and 2 runnable tasks (utilization: 6%).
[+] Try parallel jobs - see /usr/local/share/doc/afl/fuzzing_in_depth.md#c-using-multiple-cores
[*] Setting up output directories...
[*] Checking CPU core loadout...
[+] Found a free CPU core, try binding to #0.
[*] Validating target binary...
[*] Scanning '/tmp/in'...
[*] Creating hard links for all input files...
[+] Loaded a total of 1 seeds.
[*] Spinning up the NYX backend...
[!] libnyx: spawning qemu with:
/AFLplusplus/nyx_mode/QEMU-Nyx/x86_64-softmmu/qemu-system-x86_64 -kernel /AFLplusplus/nyx_mode/packer/linux_initramfs/bzImage-linux-4.15-rc7 -initrd /AFLplusplus/nyx_mode/packer/linux_initramfs/init.cpio.gz -append nokaslr oops=panic nopti ignore_rlimit_data -display none -serial none -enable-kvm -net none -k de -m 4096 -chardev socket,server,path=/tmp/out/workdir/interface_0,id=nyx_interface -device nyx,chardev=nyx_interface,bitmap_size=65536,input_buffer_size=1048576,worker_id=0,workdir=/tmp/out/workdir,sharedir=/tmp/fuzzamoto_scenario-http-server,aux_buffer_size=4096 -machine kAFL64-v1 -cpu kAFL64-Hypervisor-v1 -fast_vm_reload path=/tmp/out/workdir/snapshot/,load=off,skip_serialization=on
[QEMU-Nyx] Could not access KVM-PT kernel module!
[QEMU-Nyx] Trying vanilla KVM...
[QEMU-Nyx] NYX runs in fallback mode (no Intel-PT tracing or nested hypercall support)!
[QEMU-NYX] Max Dirty Ring Size -> 1048576 (Entries: 65536)
[QEMU-Nyx] Warning: Attempt to use unsupported CPU model (PT) without KVM-PT (Hint: use '-cpu kAFL64-Hypervisor-v2' instead)
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-vintr-pending [bit 2]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-tsc-offset [bit 3]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-hlt-exit [bit 7]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-invlpg-exit [bit 9]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-mwait-exit [bit 10]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-rdpmc-exit [bit 11]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-rdtsc-exit [bit 12]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-cr3-load-noexit [bit 15]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-cr3-store-noexit [bit 16]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-cr8-load-exit [bit 19]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-cr8-store-exit [bit 20]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-flexpriority [bit 21]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-vnmi-pending [bit 22]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-movdr-exit [bit 23]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-io-exit [bit 24]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-io-bitmap [bit 25]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-mtf [bit 27]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-msr-bitmap [bit 28]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-monitor-exit [bit 29]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-pause-exit [bit 30]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48EH).vmx-secondary-ctls [bit 31]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-xapic [bit 0]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-ept [bit 1]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-desc-exit [bit 2]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-rdtscp-exit [bit 3]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-x2apic [bit 4]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-vpid [bit 5]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-wbinvd-exit [bit 6]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-unrestricted-guest [bit 7]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-invpcid-exit [bit 12]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-vmfunc [bit 13]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-shadow-vmcs [bit 14]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-intr-exit [bit 0]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-nmi-exit [bit 3]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-vnmi [bit 5]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-preemption-timer [bit 6]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-nosave-debugctl [bit 2]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-ack-intr [bit 15]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-save-pat [bit 18]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-pat [bit 19]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-save-efer [bit 20]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-efer [bit 21]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-save-preemption-timer [bit 22]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-noload-debugctl [bit 2]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-ia32e-mode [bit 9]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-pat [bit 14]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-efer [bit 15]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(485H).vmx-store-lma [bit 5]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(485H).vmx-activity-hlt [bit 6]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(485H).vmx-vmwrite-vmexit-fields [bit 29]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-ept-execonly [bit 0]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-page-walk-4 [bit 6]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH) [bit 14]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-ept-2mb [bit 16]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-ept-1gb [bit 17]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-invept [bit 20]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-eptad [bit 21]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-invept-single-context [bit 25]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-invept-all-context [bit 26]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-invvpid [bit 32]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-invvpid-single-addr [bit 40]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-invept-single-context [bit 41]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-invvpid-all-context [bit 42]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-invept-single-context-noglobals [bit 43]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(480H).vmx-ins-outs [bit 54]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(480H).vmx-true-ctls [bit 55]
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(491H).vmx-eptp-switching [bit 0]
[QEMU-NYX] Dirty ring mmap region located at 0x7fdbb8b65000
[QEMU-NYX] Booting VM to start fuzzing...
[!] libnyx: input buffer is write protected
[hget] 16480 bytes received from hypervisor! (hcat_no_pt)
[hget] 16432 bytes received from hypervisor! (habort_no_pt)
[hget] 172797480 bytes received from hypervisor! (bitcoind)
[hget] 215000 bytes received from hypervisor! (ld-linux-x86-64.so.2)
[hget] 1922136 bytes received from hypervisor! (libc.so.6)
[hget] 125312 bytes received from hypervisor! (libgcc_s.so.1)
[hget] 911904 bytes received from hypervisor! (libm.so.6)
[hget] 16752 bytes received from hypervisor! (libnyx_crash_handler.so)
[hget] 60328 bytes received from hypervisor! (libresolv.so.2)
[hget] 2190440 bytes received from hypervisor! (libstdc++.so.6)
[hget] 16855544 bytes received from hypervisor! (scenario-http-server)
[hcat] 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
[hcat] Illegal instruction
[!] libnyx failed to initialize QEMU-Nyx: agent abort() ->
USER_ABORT called: target has terminated without initializing the fuzzing agent ...
[-] PROGRAM ABORT : Something went wrong ...
Location : afl_fsrv_start(), src/afl-forkserver.c:764
For reference, this seems to fail when running ./bitcoind_proxy here:
fuzzamoto/fuzzamoto-cli/src/main.rs
Lines 214 to 220 in d0f8670
I'll try to debug this. Last I checked, I was able to run fuzzamoto without using docker.