From 4b1ee12811c7517217f768636fe4383a5f4e115f Mon Sep 17 00:00:00 2001 From: NimbleAINinja Date: Thu, 17 Sep 2026 23:34:18 -0400 Subject: [PATCH] Announce the VM ready only once its QMP monitor answers Launching intermittently failed with "The virtual machine started, but Try Omarchy could not enable safe Mac sleep ... cannot connect to QMP socket: Connection refused", most often on the first launch after installing a build, and the guest console log was empty afterwards. run-qemu-gpu.sh printed "Ready. QMP:" as soon as QEMU had created its socket files. QEMU creates those early in initialisation, but accepts monitor connections only from its main loop, which starts when initialisation finishes, and the socket listens with a backlog of one. The helper connected on Ready and waited two seconds for a greeting. On a slow start -- an 8 GiB guest, VirGL and ANGLE setup, cold caches after an install -- initialisation outlasted that wait. The connection had been queued by the kernel, and a queued connection stays queued when its client gives up, so the backlog was now full and every later connect was refused until the main loop ran. The helper reads that as a broken monitor and sends QEMU SIGTERM, which is why the console log was empty. The mechanism was reproduced against the shipped QEMU with a deliberately slow initialisation: one probe that gave up, four refused retries, then success the instant the main loop came up. Spacing the helper's retries cannot help; they retry into a full queue. Make Ready mean what its consumer assumes. A new sourced library waits until a connection to the QMP socket receives a greeting, retrying a refused connect or a silent monitor every 100 ms. A probe that gives up may itself occupy the backlog, and the main loop drains it as its first act, so the next probe gets through. It fails if QEMU exits or after 60 seconds, so a monitor that never answers still fails startup. The test drives the function against a stand-in that listens immediately but accepts only after a delay, the way QEMU does, and checks that Ready waits for the answer, does not hold up a monitor that answers at once, and reports a dead QEMU promptly; it also checks that the launcher consults the monitor before printing the Ready line. The two launcher contract tests stage the new library into their bundle and their fake QEMU now answers the monitor with a greeting, as QEMU does. Co-Authored-By: Claude Fable 5.1 --- Makefile | 1 + macos/Tests/qemu-memory-contract.test.sh | 23 +++++ macos/Tests/qemu-monitor-ready.test.sh | 110 ++++++++++++++++++++++ macos/Tests/run-qemu-ssh-contract.test.sh | 23 +++++ macos/build-app.sh | 2 + macos/qemu-monitor-ready.sh | 56 +++++++++++ macos/run-qemu-gpu.sh | 10 ++ 7 files changed, 225 insertions(+) create mode 100755 macos/Tests/qemu-monitor-ready.test.sh create mode 100644 macos/qemu-monitor-ready.sh diff --git a/Makefile b/Makefile index 223f993..311f92e 100644 --- a/Makefile +++ b/Makefile @@ -69,6 +69,7 @@ test: @$(ROOT)/macos/Tests/run-qemu-ssh-contract.test.sh @$(ROOT)/macos/Tests/qemu-memory-contract.test.sh @$(ROOT)/macos/Tests/qemu-power-actions.test.sh + @$(ROOT)/macos/Tests/qemu-monitor-ready.test.sh @$(ROOT)/macos/Tests/qemu-persistent-storage.test.sh @PYTHONDONTWRITEBYTECODE=1 python3 "$(ROOT)/macos/Tests/resize-vm-disk.test.py" diff --git a/macos/Tests/qemu-memory-contract.test.sh b/macos/Tests/qemu-memory-contract.test.sh index f14debe..2e15e4c 100755 --- a/macos/Tests/qemu-memory-contract.test.sh +++ b/macos/Tests/qemu-memory-contract.test.sh @@ -44,8 +44,10 @@ mkdir -p \ /bin/cp "$macos_dir/run-qemu-gpu.sh" "$resources/scripts/run-qemu-gpu.sh" /bin/cp "$macos_dir/qemu-port-forwarding.sh" "$resources/scripts/qemu-port-forwarding.sh" /bin/cp "$macos_dir/qemu-networking.sh" "$resources/scripts/qemu-networking.sh" +/bin/cp "$macos_dir/qemu-monitor-ready.sh" "$resources/scripts/qemu-monitor-ready.sh" chmod 755 "$resources/scripts/run-qemu-gpu.sh" chmod 644 "$resources/scripts/qemu-port-forwarding.sh" +chmod 644 "$resources/scripts/qemu-monitor-ready.sh" cat >"$contents/MacOS/omarchy-vm-helper" <<'SH' #!/bin/bash @@ -104,10 +106,16 @@ import os from pathlib import Path import socket import sys +import threading import time arguments = sys.argv[1:] Path(os.environ["FAKE_QEMU_LOG"]).write_text("\n".join(arguments) + "\n") +qmp_paths = { + arguments[index + 1][5:].split(",", 1)[0] + for index, argument in enumerate(arguments[:-1]) + if argument == "-qmp" and arguments[index + 1].startswith("unix:") +} socket_paths = [] for argument in arguments: if argument.startswith("unix:"): @@ -128,6 +136,21 @@ if os.environ.get("FAKE_QEMU_SKIP_SOCKETS") != "1": server.bind(path) server.listen(1) servers.append(server) + # Like QEMU, answer the QMP monitor with a greeting; the launcher + # waits for that before declaring the VM ready. + if path in qmp_paths: + def greet(server=server): + while True: + try: + client, _ = server.accept() + except OSError: + return + try: + client.sendall(b'{"QMP": {"version": {}, "capabilities": []}}\r\n') + except OSError: + pass + client.close() + threading.Thread(target=greet, daemon=True).start() time.sleep(float(os.environ.get("FAKE_QEMU_LIFETIME", "0.20"))) for server in servers: diff --git a/macos/Tests/qemu-monitor-ready.test.sh b/macos/Tests/qemu-monitor-ready.test.sh new file mode 100755 index 0000000..815d0d4 --- /dev/null +++ b/macos/Tests/qemu-monitor-ready.test.sh @@ -0,0 +1,110 @@ +#!/bin/bash + +# The launcher's "Ready. QMP:" line is a contract: the helper connects to that +# socket and tears the VM down if the monitor does not answer. QEMU creates the +# socket file early in its initialisation but only accepts connections once its +# main loop runs, with a listen backlog of one, so a client that connects too +# early and gives up leaves every later connect refused until the loop starts. +# Ready must therefore mean "the monitor answered", not "the file exists". + +set -euo pipefail + +test_dir=$(cd "$(dirname "$0")" && pwd -P) +macos_dir=$(cd "$test_dir/.." && pwd -P) +library="$macos_dir/qemu-monitor-ready.sh" +launcher="$macos_dir/run-qemu-gpu.sh" + +fail() { + printf 'qemu-monitor-ready.test: %s\n' "$*" >&2 + exit 1 +} + +[[ -f $library ]] || fail "missing library: $library" +# shellcheck source=../qemu-monitor-ready.sh +source "$library" +declare -F qemu_wait_for_qmp_monitor >/dev/null || \ + fail 'library must define qemu_wait_for_qmp_monitor' + +scratch=$(mktemp -d "${TMPDIR:-/tmp}/qemu-monitor-ready.XXXXXX") +cleanup() { + [[ -z ${server_pid:-} ]] || kill "$server_pid" 2>/dev/null || true + [[ -z ${sleeper_pid:-} ]] || kill "$sleeper_pid" 2>/dev/null || true + rm -rf "$scratch" +} +trap cleanup EXIT + +# A stand-in for QEMU: binds and listens (backlog 1) immediately, like the +# chardev does during init, but only starts accepting after ACCEPT_DELAY +# seconds, like the main loop. Each accepted client gets a QMP greeting. +fake_qemu() { + python3 - "$1" "$2" <<'PY' & +import socket, sys, time +path, delay = sys.argv[1], float(sys.argv[2]) +server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) +server.bind(path) +server.listen(1) +time.sleep(delay) +while True: + client, _ = server.accept() + # A probe that gave up during the delay is still queued here; like QEMU, + # shrug off the peer having gone away. + try: + client.sendall(b'{"QMP": {"version": {}, "capabilities": []}}\r\n') + except OSError: + pass + client.close() +PY + server_pid=$! +} + +# Something with a live pid to stand in for the QEMU process. +sleep 300 & +sleeper_pid=$! + +# 1. Slow init: the monitor answers only after 1.5 s. Ready must wait for it. +socket_path="$scratch/slow.sock" +fake_qemu "$socket_path" 1.5 +until [[ -S $socket_path ]]; do sleep 0.02; done +start=$(python3 -c 'import time; print(time.monotonic())') +qemu_wait_for_qmp_monitor "$socket_path" "$sleeper_pid" || \ + fail 'a monitor that answers after a slow init must count as ready' +elapsed=$(python3 -c "import time; print(time.monotonic() - $start)") +python3 -c "import sys; sys.exit(0 if $elapsed >= 1.4 else 1)" || \ + fail "ready was declared after ${elapsed}s, before the monitor could answer" +kill "$server_pid"; wait "$server_pid" 2>/dev/null || true; unset server_pid + +# 2. Fast init: an immediately answering monitor must not be held up. +socket_path="$scratch/fast.sock" +fake_qemu "$socket_path" 0 +until [[ -S $socket_path ]]; do sleep 0.02; done +start=$(python3 -c 'import time; print(time.monotonic())') +qemu_wait_for_qmp_monitor "$socket_path" "$sleeper_pid" || \ + fail 'an answering monitor must count as ready' +elapsed=$(python3 -c "import time; print(time.monotonic() - $start)") +python3 -c "import sys; sys.exit(0 if $elapsed < 1.0 else 1)" || \ + fail "an answering monitor took ${elapsed}s to be declared ready" +kill "$server_pid"; wait "$server_pid" 2>/dev/null || true; unset server_pid + +# 3. QEMU gone: a socket file with no process behind it must fail promptly +# rather than wait out the whole deadline. +socket_path="$scratch/dead.sock" +python3 -c 'import socket, sys; s = socket.socket(socket.AF_UNIX); s.bind(sys.argv[1])' "$socket_path" +kill "$sleeper_pid"; wait "$sleeper_pid" 2>/dev/null || true +dead_pid=$sleeper_pid; unset sleeper_pid +start=$(python3 -c 'import time; print(time.monotonic())') +if qemu_wait_for_qmp_monitor "$socket_path" "$dead_pid" 2>/dev/null; then + fail 'a monitor whose QEMU has exited must not count as ready' +fi +elapsed=$(python3 -c "import time; print(time.monotonic() - $start)") +python3 -c "import sys; sys.exit(0 if $elapsed < 3.0 else 1)" || \ + fail "a dead QEMU took ${elapsed}s to be reported" + +# 4. The launcher must consult the monitor before announcing Ready. +wait_line=$(grep -n 'qemu_wait_for_qmp_monitor "$qmp_socket" "$qemu_pid"' "$launcher" | cut -d: -f1 | head -1) +ready_line=$(grep -n '^echo "\[qemu-gpu\] Ready. QMP: \$qmp_socket" >&2$' "$launcher" | cut -d: -f1 | head -1) +[[ -n $wait_line && -n $ready_line ]] || \ + fail 'launcher must wait for the QMP monitor and print the Ready line' +(( wait_line < ready_line )) || \ + fail 'launcher must wait for the QMP monitor before printing Ready' + +echo 'qemu-monitor-ready.test: PASS' diff --git a/macos/Tests/run-qemu-ssh-contract.test.sh b/macos/Tests/run-qemu-ssh-contract.test.sh index f3f7a7f..4a89fd2 100755 --- a/macos/Tests/run-qemu-ssh-contract.test.sh +++ b/macos/Tests/run-qemu-ssh-contract.test.sh @@ -49,8 +49,10 @@ mkdir -p \ /bin/cp "$macos_dir/run-qemu-gpu.sh" "$resources/scripts/run-qemu-gpu.sh" /bin/cp "$macos_dir/qemu-port-forwarding.sh" "$resources/scripts/qemu-port-forwarding.sh" /bin/cp "$macos_dir/qemu-networking.sh" "$resources/scripts/qemu-networking.sh" +/bin/cp "$macos_dir/qemu-monitor-ready.sh" "$resources/scripts/qemu-monitor-ready.sh" chmod 755 "$resources/scripts/run-qemu-gpu.sh" chmod 644 "$resources/scripts/qemu-port-forwarding.sh" +chmod 644 "$resources/scripts/qemu-monitor-ready.sh" cat >"$contents/MacOS/omarchy-vm-helper" <<'SH' #!/bin/bash @@ -114,6 +116,7 @@ import os from pathlib import Path import socket import sys +import threading import time arguments = sys.argv[1:] @@ -137,6 +140,11 @@ if is_recovery: 'console=tty0 console=hvc0 loglevel=3"}}\n' ) export_path.joinpath("complete").write_text("try-omarchy-boot-export-v1\n") +qmp_paths = { + arguments[index + 1][5:].split(",", 1)[0] + for index, argument in enumerate(arguments[:-1]) + if argument == "-qmp" and arguments[index + 1].startswith("unix:") +} socket_paths = [] for argument in arguments: if argument.startswith("unix:"): @@ -157,6 +165,21 @@ if os.environ.get("FAKE_QEMU_SKIP_SOCKETS") != "1": server.bind(path) server.listen(1) servers.append(server) + # Like QEMU, answer the QMP monitor with a greeting; the launcher + # waits for that before declaring the VM ready. + if path in qmp_paths: + def greet(server=server): + while True: + try: + client, _ = server.accept() + except OSError: + return + try: + client.sendall(b'{"QMP": {"version": {}, "capabilities": []}}\r\n') + except OSError: + pass + client.close() + threading.Thread(target=greet, daemon=True).start() time.sleep(float(os.environ.get("FAKE_QEMU_LIFETIME", "0.20"))) for server in servers: diff --git a/macos/build-app.sh b/macos/build-app.sh index 9b79531..429af5b 100755 --- a/macos/build-app.sh +++ b/macos/build-app.sh @@ -189,6 +189,8 @@ install -m 0644 "$macos_dir/qemu-persistent-storage.sh" \ install -m 0644 "$macos_dir/qemu-port-forwarding.sh" \ "$contents/Resources/scripts/qemu-port-forwarding.sh" install -m 0644 "$macos_dir/qemu-networking.sh" "$contents/Resources/scripts/qemu-networking.sh" +install -m 0644 "$macos_dir/qemu-monitor-ready.sh" \ + "$contents/Resources/scripts/qemu-monitor-ready.sh" for guest_resource in \ LICENSE.omarchy \ SHA256SUMS \ diff --git a/macos/qemu-monitor-ready.sh b/macos/qemu-monitor-ready.sh new file mode 100644 index 0000000..99b78ca --- /dev/null +++ b/macos/qemu-monitor-ready.sh @@ -0,0 +1,56 @@ +# Wait until QEMU's QMP monitor actually answers. +# +# QEMU creates its chardev socket files early in initialisation, but the +# monitor only accepts connections once the main loop runs, and the socket +# listens with a backlog of one. A client that connects during that gap and +# gives up leaves its connection queued until QEMU accepts it, so every later +# connect is refused until the main loop starts. Announcing the VM as ready on +# the socket file alone therefore hands the helper a socket it may not be able +# to use for seconds; the helper treats that as a broken monitor and tears the +# VM down. Ready has to mean the monitor answered. +# +# Sourced by run-qemu-gpu.sh; expects `fail` to be defined by the caller. + +# qemu_wait_for_qmp_monitor SOCKET QEMU_PID +# +# Returns 0 once a connection to SOCKET receives a QMP greeting. Each probe +# gives the monitor 250 ms to answer and then disconnects; a refused connect +# or a silent monitor is retried after 100 ms. Both outcomes are expected +# while QEMU is still initialising: a probe that gave up may itself occupy the +# backlog, and the main loop drains it as its first act, so the next probe +# gets through. Fails if QEMU exits, or after 60 s so a monitor that never +# answers still fails startup instead of hanging it. +qemu_wait_for_qmp_monitor() { + local socket_path=$1 + local qemu_pid=$2 + local deadline=$((SECONDS + 60)) + + while (( SECONDS < deadline )); do + if python3 - "$socket_path" <<'PY' 2>/dev/null +import socket +import sys + +client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) +client.settimeout(0.25) +try: + client.connect(sys.argv[1]) + greeting = client.recv(4096) +except (OSError, socket.timeout): + raise SystemExit(1) +finally: + client.close() +raise SystemExit(0 if greeting.lstrip().startswith(b'{"QMP"') else 1) +PY + then + return 0 + fi + kill -0 "$qemu_pid" 2>/dev/null || { + echo 'qemu-monitor-ready: QEMU exited before its QMP monitor answered' >&2 + return 1 + } + sleep 0.1 + done + + echo 'qemu-monitor-ready: the QMP monitor did not answer within 60 seconds' >&2 + return 1 +} diff --git a/macos/run-qemu-gpu.sh b/macos/run-qemu-gpu.sh index ae15ffd..6c7701f 100755 --- a/macos/run-qemu-gpu.sh +++ b/macos/run-qemu-gpu.sh @@ -52,6 +52,7 @@ qemu_bin="$resources_dir/runtime/bin/Try Omarchy" native_bridge="$contents_dir/MacOS/omarchy-vm-helper" storage_library="$script_dir/qemu-persistent-storage.sh" port_forwarding_library="$script_dir/qemu-port-forwarding.sh" +monitor_ready_library="$script_dir/qemu-monitor-ready.sh" [[ $(uname -m) == arm64 ]] || fail "requires an ARM64 Mac" [[ $(uname -s) == Darwin ]] || fail "requires macOS" @@ -931,6 +932,9 @@ fi [[ -f $port_forwarding_library && ! -L $port_forwarding_library ]] || { fail "port-forwarding library is missing or unsafe: $port_forwarding_library" } +[[ -f $monitor_ready_library && ! -L $monitor_ready_library ]] || { + fail "monitor-readiness library is missing or unsafe: $monitor_ready_library" +} # These libraries are sealed resources in normal app launches. The complete # app bundle was verified above before either file can execute. Inspect-only is @@ -939,6 +943,8 @@ fi source "$storage_library" # shellcheck source=qemu-port-forwarding.sh source "$port_forwarding_library" +# shellcheck source=qemu-monitor-ready.sh +source "$monitor_ready_library" source "$script_dir/qemu-networking.sh" qemu_network_validate if [[ $QEMU_NETWORK_MODE == bridged ]]; then @@ -1644,6 +1650,10 @@ done [[ -S $authentication_bridge_socket ]] || fail "QEMU did not create its private authentication bridge socket" [[ -S $camera_bridge_socket ]] || fail "QEMU did not create its private camera bridge socket" [[ -S $clipboard_bridge_socket ]] || fail "QEMU did not create its private clipboard bridge socket" +# The socket file appears before QEMU's main loop accepts connections, and the +# helper tears the VM down if the monitor behind this line does not answer. +qemu_wait_for_qmp_monitor "$qmp_socket" "$qemu_pid" || \ + fail "QEMU's QMP monitor did not become ready" echo "[qemu-gpu] Ready. QMP: $qmp_socket" >&2 # FD 9 deliberately remains open only in QEMU. Letting the sibling audio