Add e2e + manual QA test harness

CLAUDE.md

@@ -41,11 +41,31 @@ Local-first text-to-speech desktop app. Kokoro 82M model runs entirely on-device
 
 ## Testing
 
-- **Install dev deps**: `pip install -r requirements-dev.txt` (pytest + fastapi + httpx; no torch/kokoro needed)
-- **Run**: `python -m pytest` from the repo root
-- The suite mocks the heavy ML/audio stack (numpy, sounddevice, soundfile, kokoro) via fixtures in `tests/conftest.py`, so it runs on any Python 3.10-3.12 without a model download or audio hardware.
-- Coverage: config load/merge/corruption, the Kokoro `speak()` flow, audio helpers, every REST endpoint, the CORS policy (regression guard for issue 002), API-key redaction, and the OpenAI/ElevenLabs fallbacks.
-- CI runs the suite on push/PR across Python 3.10-3.12 (`.github/workflows/tests.yml`).
+Three layers, fastest first:
+
+1. **Unit / integration** (`tests/test_*.py`) — in-process via Starlette
+   `TestClient`. Mocks the heavy ML/audio stack (numpy, sounddevice, soundfile,
+   kokoro) via fixtures in `tests/conftest.py`. Covers config
+   load/merge/corruption, the Kokoro `speak()` flow, audio helpers, every REST
+   endpoint, the CORS policy (regression guard for issue 002), API-key
+   redaction, and the OpenAI/ElevenLabs fallbacks.
+2. **Live end-to-end** (`tests/e2e/`) — boots the real ASGI app under uvicorn on
+   an ephemeral port and drives it over a real socket (stdlib HTTP client).
+   Proves routing/redirects, CORS behaviour over the wire, and that
+   `PATCH /config` round-trips secrets to a real file while redacting the
+   response. Audio/model leaf calls are stubbed, so it still needs no torch/kokoro.
+3. **Manual / dev QA** (`docs/QA.md` + `scripts/qa_probe.py`) — for what CI
+   can't do: real audio, the model download, live API keys, and the Chrome
+   extension. `qa_probe.py` drives a *running* daemon (`python main.py`).
+
+Commands:
+
+- **Install dev deps**: `pip install -r requirements-dev.txt` (pytest, fastapi, httpx, uvicorn; no torch/kokoro)
+- **Everything**: `python -m pytest`
+- **Fast unit only**: `python -m pytest -m "not e2e"`
+- **Live e2e only**: `python -m pytest tests/e2e`
+- **Subprocess liveness smoke**: `scripts/smoke.sh`
+- CI runs the full suite (layers 1 + 2) on push/PR across Python 3.10-3.12 (`.github/workflows/tests.yml`).
 
 ## Deployment
 

docs/QA.md

@@ -0,0 +1,86 @@
+# ReadOut — Manual QA Checklist
+
+Automated tests cover the HTTP contract, config logic, CORS policy, and engine
+dispatch (see [Testing](../CLAUDE.md#testing)). They deliberately stub the
+parts that need a human or real hardware. This checklist covers **only those
+parts**: real audio, the model download, engine switching with live keys, and
+the Chrome extension.
+
+Run these on a real dev machine against a real daemon (`python main.py`), not
+in CI. The `scripts/qa_probe.py` driver talks to the running daemon for you.
+
+---
+
+## 0. Pre-flight
+
+- [ ] `pip install -r requirements.txt` (full runtime stack incl. kokoro/torch)
+- [ ] `brew install espeak-ng` (macOS) — required by Kokoro
+- [ ] Start the daemon: `python main.py`
+- [ ] In another shell: `python scripts/qa_probe.py status` → `HTTP 200`, `"status"` is `loading` or `ready`
+
+## 1. First-run model download (Kokoro)
+
+- [ ] On a machine with no `~/.readout/.model_ready`, start the daemon
+- [ ] `qa_probe.py status` reports `"status": "loading"` and `"model_ready": false` initially
+- [ ] After the ~300 MB download + load, status flips to `"ready"` / `"model_ready": true`
+- [ ] `~/.readout/.model_ready` now exists; a restart comes up `ready` immediately
+
+## 2. Real audio playback (Kokoro)
+
+- [ ] `qa_probe.py speak "Hello from ReadOut"` → audio plays through the default output
+- [ ] `--voice af_sky` / `--voice am_adam` audibly change the voice
+- [ ] `--speed 1.5` is faster, `--speed 0.75` slower
+- [ ] While audio is playing, `qa_probe.py stop` (or a new `speak`) halts it promptly
+
+## 3. File save
+
+- [ ] `qa_probe.py speak "save test" --save` → response has `saved_to`; a `.wav` exists there
+- [ ] Set `always_save` (`qa_probe.py config` is keys-only; edit `~/.readout/config.json` or use the control panel) → every `speak` writes a file
+- [ ] Default save dir is `~/Desktop/ReadOut` unless `save_dir` is overridden
+
+## 4. Engine switching (OpenAI / ElevenLabs) — needs live keys
+
+- [ ] `qa_probe.py config --engine openai --openai-key sk-...` → `HTTP 200`, response shows `"openai_api_key": "***"` (redacted) and the command prints no warning
+- [ ] `~/.readout/config.json` contains the **real** key (redaction is response-only)
+- [ ] `qa_probe.py speak "openai voice test" --voice nova` → OpenAI audio plays; `stop` halts it
+- [ ] Switch to `--engine elevenlabs --elevenlabs-key ...`; `speak` plays ElevenLabs audio
+      - Note: ElevenLabs streams MP3; if playback fails with a decode error, the
+        local `soundfile`/`libsndfile` build lacks MP3 support (needs ≥ 1.1) — see issue 003 context
+- [ ] A bad key surfaces a clean `{"status":"error","message":"... 401 ..."}`, not a stack trace
+- [ ] Switch back: `qa_probe.py config --engine kokoro`
+
+## 5. Control panel (served by the daemon)
+
+- [ ] Open `http://127.0.0.1:7778/` → redirects to `/control`, panel renders
+- [ ] The panel can change voice/speed/engine and trigger speak/stop
+- [ ] The panel does **not** display stored API keys (the server never sends them back)
+
+## 6. Chrome extension
+
+- [ ] Load unpacked: `chrome://extensions` → Developer mode → Load unpacked → `extension/`
+- [ ] Select text on any page → right-click → "Read aloud via ReadOut" → audio plays
+- [ ] Toolbar popup: status indicator shows daemon up/down; Play/Stop work; settings persist
+- [ ] With the daemon stopped, the popup shows an unreachable/error state (no crash)
+
+## 7. Security smoke (the issue-002 regression)
+
+- [ ] `python scripts/qa_probe.py cors --origin https://evil.com` → prints **BLOCKED** (exit 0)
+- [ ] `... cors --origin chrome-extension://abcdefghijklmnopabcdefghijklmnop` → prints **ALLOWED**
+- [ ] In a random website's DevTools console, `fetch('http://127.0.0.1:7778/status')` is blocked by CORS
+- [ ] `qa_probe.py config --openai-key sk-canary` then confirm `sk-canary` never appears in the HTTP response (the driver warns loudly if it does)
+- [ ] Known gap (issue 003): a cross-origin page **can** still fire `POST /stop` as a side effect — verify impact is limited to halting playback
+
+## 8. Platform: macOS 26 Tk skip (issue 001)
+
+- [ ] On macOS 26+, the daemon + tray run without launching the Tk window (no `GetRGBA`/NSApplication crash)
+- [ ] On macOS < 26 / other platforms, the Tk window still appears as before
+
+---
+
+### Quick automated cross-checks (optional, not a substitute for the above)
+
+```bash
+scripts/smoke.sh                 # boots the real app in a subprocess, probes liveness + CORS
+python -m pytest tests/e2e -v    # live-server harness over a real socket (audio stubbed)
+python -m pytest -m "not e2e"    # fast in-process unit/integration suite only
+```

pytest.ini

@@ -3,5 +3,7 @@ testpaths = tests
 python_files = test_*.py
 python_functions = test_*
 addopts = -ra -q
+markers =
+    e2e: live-server end-to-end tests (boot uvicorn on a real socket; need the uvicorn dev dep)
 filterwarnings =
     ignore::DeprecationWarning

requirements-dev.txt

@@ -6,4 +6,4 @@
 pytest>=7.0
 fastapi
 httpx<0.28        # starlette TestClient transport
-uvicorn           # live source HTTP smoke / release_preflight -RunSourceSmoke
+uvicorn           # tests/e2e + live source HTTP smoke / release_preflight -RunSourceSmoke

scripts/qa_probe.py

@@ -0,0 +1,142 @@
+#!/usr/bin/env python3
+"""qa_probe.py — manual QA driver for a RUNNING ReadOut daemon.
+
+Unlike the pytest suites (which stub audio/model), this talks to the real
+daemon you started with `python main.py`, so `speak` plays real audio through
+the real engine. Use it to sanity-check a dev build or a packaged .app, and to
+verify the CORS lockdown from the command line.
+
+Stdlib only — no install needed. Run `python main.py` first, then:
+
+    python scripts/qa_probe.py status
+    python scripts/qa_probe.py voices
+    python scripts/qa_probe.py speak "Hello from ReadOut" --voice af_sky --speed 1.1
+    python scripts/qa_probe.py stop
+    python scripts/qa_probe.py config --engine openai --openai-key sk-...
+    python scripts/qa_probe.py cors --origin https://evil.com   # expect: BLOCKED
+
+Add --url http://127.0.0.1:7778 to target a non-default host/port.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import urllib.error
+import urllib.request
+
+DEFAULT_URL = "http://127.0.0.1:7778"
+
+
+def _call(url, path, method="GET", body=None, headers=None):
+    data = json.dumps(body).encode() if body is not None else None
+    h = dict(headers or {})
+    if data:
+        h.setdefault("Content-Type", "application/json")
+    req = urllib.request.Request(url + path, data=data, headers=h, method=method)
+    try:
+        r = urllib.request.urlopen(req, timeout=15)
+        return r.status, {k.lower(): v for k, v in r.getheaders()}, r.read().decode()
+    except urllib.error.HTTPError as e:
+        return e.code, {k.lower(): v for k, v in e.headers.items()}, e.read().decode()
+    except urllib.error.URLError as e:
+        print(f"ERROR: cannot reach {url} — is the daemon running? ({e.reason})", file=sys.stderr)
+        raise SystemExit(2)
+
+
+def _show(status, body):
+    print(f"HTTP {status}")
+    try:
+        print(json.dumps(json.loads(body), indent=2))
+    except json.JSONDecodeError:
+        print(body[:500])
+
+
+def cmd_status(a):
+    _show(*_call(a.url, "/status")[::2])
+
+
+def cmd_voices(a):
+    _show(*_call(a.url, "/voices")[::2])
+
+
+def cmd_speak(a):
+    body = {"text": a.text, "save": a.save}
+    if a.voice:
+        body["voice"] = a.voice
+    if a.speed:
+        body["speed"] = a.speed
+    _show(*_call(a.url, "/speak", "POST", body)[::2])
+
+
+def cmd_stop(a):
+    _show(*_call(a.url, "/stop", "POST")[::2])
+
+
+def cmd_config(a):
+    body = {}
+    if a.engine:
+        body["engine"] = a.engine
+    if a.voice:
+        body["voice"] = a.voice
+    if a.openai_key is not None:
+        body["openai_api_key"] = a.openai_key
+    if a.elevenlabs_key is not None:
+        body["elevenlabs_api_key"] = a.elevenlabs_key
+    if not body:
+        print("Nothing to update. Pass --engine/--voice/--openai-key/--elevenlabs-key.")
+        return
+    status, _, resp = _call(a.url, "/config", "PATCH", body)
+    _show(status, resp)
+    # Defensive check: the response must never echo a plaintext key.
+    for secret in (a.openai_key, a.elevenlabs_key):
+        if secret and secret in resp:
+            print("\n*** WARNING: a plaintext API key appeared in the response! ***")
+
+
+def cmd_cors(a):
+    """Send an Origin header and report whether CORS would allow it."""
+    status, headers, _ = _call(a.url, "/status", headers={"Origin": a.origin})
+    acao = headers.get("access-control-allow-origin")
+    if acao == a.origin:
+        print(f"ALLOWED  — server echoed access-control-allow-origin: {acao}")
+    elif acao == "*":
+        print("WILDCARD — server returned '*' (this is the issue-002 regression!)")
+        raise SystemExit(1)
+    else:
+        print(f"BLOCKED  — no matching access-control-allow-origin for {a.origin!r}")
+
+
+def main(argv=None):
+    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    p.add_argument("--url", default=DEFAULT_URL, help=f"daemon base URL (default {DEFAULT_URL})")
+    sub = p.add_subparsers(dest="cmd", required=True)
+
+    sub.add_parser("status").set_defaults(func=cmd_status)
+    sub.add_parser("voices").set_defaults(func=cmd_voices)
+    sub.add_parser("stop").set_defaults(func=cmd_stop)
+
+    sp = sub.add_parser("speak")
+    sp.add_argument("text")
+    sp.add_argument("--voice")
+    sp.add_argument("--speed", type=float)
+    sp.add_argument("--save", action="store_true")
+    sp.set_defaults(func=cmd_speak)
+
+    cp = sub.add_parser("config")
+    cp.add_argument("--engine", choices=["kokoro", "openai", "elevenlabs"])
+    cp.add_argument("--voice")
+    cp.add_argument("--openai-key", dest="openai_key")
+    cp.add_argument("--elevenlabs-key", dest="elevenlabs_key")
+    cp.set_defaults(func=cmd_config)
+
+    cc = sub.add_parser("cors")
+    cc.add_argument("--origin", required=True, help="e.g. https://evil.com")
+    cc.set_defaults(func=cmd_cors)
+
+    args = p.parse_args(argv)
+    args.func(args)
+
+
+if __name__ == "__main__":
+    main()

scripts/smoke.sh

@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+# smoke.sh — boot the real ASGI app in a subprocess and probe it over HTTP.
+#
+# Complements the pytest e2e harness (which runs uvicorn in a thread): this
+# proves the app starts as a *separate process* via the same entrypoint used in
+# production, with an isolated HOME so it never touches your real ~/.readout.
+# Requires the runtime deps (fastapi + uvicorn); does NOT need torch/kokoro
+# because /status and /voices don't load the model.
+#
+# Usage: scripts/smoke.sh [port]   (default 7799)
+set -euo pipefail
+
+PORT="${1:-7799}"
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+TMPHOME="$(mktemp -d)"
+export HOME="$TMPHOME"        # isolate ~/.readout for this run
+
+cleanup() {
+  [ -n "${SRV_PID:-}" ] && kill "$SRV_PID" 2>/dev/null || true
+  rm -rf "$TMPHOME"
+}
+trap cleanup EXIT
+
+cd "$ROOT"
+python -m uvicorn server:app --host 127.0.0.1 --port "$PORT" --log-level warning &
+SRV_PID=$!
+
+base="http://127.0.0.1:$PORT"
+ready=""
+for _ in $(seq 1 50); do
+  if curl -sf "$base/status" >/dev/null 2>&1; then ready=1; break; fi
+  sleep 0.2
+done
+[ -n "$ready" ] || { echo "FAIL: server did not become ready on $base"; exit 1; }
+
+curl -sf "$base/status" | grep -q '"version"' && echo "OK   /status"
+curl -sf "$base/voices" | grep -q '"voices"'  && echo "OK   /voices"
+
+# CORS regression guard: a hostile origin must NOT be echoed back.
+acao="$(curl -sf -D - -o /dev/null -H 'Origin: https://evil.com' "$base/status" \
+        | tr -d '\r' | awk -F': ' 'tolower($1)=="access-control-allow-origin"{print $2}')"
+if [ "$acao" = "*" ] || [ "$acao" = "https://evil.com" ]; then
+  echo "FAIL: hostile origin allowed by CORS (got '$acao')"; exit 1
+fi
+echo "OK   CORS rejects hostile origin"
+
+echo "SMOKE PASSED"