usertest: voice-moderated sessions with a Claude moderator

usertest/Caddyfile

@@ -9,8 +9,22 @@
 
 # atomic-server (server.sh), plain HTTP on loopback. It builds its links from
 # ATOMIC_DOMAIN and reads the scheme from X-Forwarded-Proto, which Caddy sets.
+# /usertest/ is the moderated-session page (page/) and /usertest/api/ its
+# moderator (moderator/). They share the data-browser's origin so the page can
+# set the test catalog in the browser's storage.
 plugins.{$BASE_DOMAIN} {
-	reverse_proxy 127.0.0.1:8080
+	redir /usertest /usertest/?{query} 308
+	handle_path /usertest/api/* {
+		reverse_proxy 127.0.0.1:8082
+	}
+	handle_path /usertest/* {
+		root * /opt/usertest/page
+		header Cache-Control "no-cache"
+		file_server
+	}
+	handle {
+		reverse_proxy 127.0.0.1:8080
+	}
 }
 
 # The test catalog and the drive app modules it points at (catalog.mjs).

usertest/README.md

@@ -11,6 +11,7 @@ rebuild it.
 | `https://plugins.<base-domain>` | atomic-server, the published e2e image of the pinned commit (`server.sh`) |
 | `https://catalog.<base-domain>/catalog.json` | the test catalog and the app modules it points at (`catalog.mjs`) |
 | `https://logs.<base-domain>` | the log collector (`collector/`) |
+| `https://plugins.<base-domain>/usertest/` | the moderated-session page (`page/`) and its voice moderator (`moderator/`, at `/usertest/api/`) |
 | `https://localthought.io` | the integration proxy, shared with everyone else (not on the droplet) |
 
 The base domain is currently `178-62-223-35.sslip.io`. sslip.io resolves
@@ -60,6 +61,7 @@ for a in calendar money notion timesheets; do (cd integrations/$a && pnpm instal
 USERTEST_LOG_URL=https://logs.178-62-223-35.sslip.io/log node usertest/catalog.mjs
 sh usertest/deploy.sh root@178.62.223.35
 ssh root@178.62.223.35 sh /opt/usertest/collector/run.sh
+ssh root@178.62.223.35 sh /opt/usertest/moderator/run.sh   # needs /etc/anthropic.env
 ssh root@178.62.223.35 sh /opt/usertest/server.sh 178-62-223-35.sslip.io
 ```
 
@@ -74,16 +76,47 @@ ssh root@178.62.223.35 sh /opt/usertest/server.sh 178-62-223-35.sslip.io
 Never rebuild into an existing version: the host refuses a module whose bytes
 no longer match the hash in the catalog.
 
-## For testers
+## Moderated sessions
+
+A tester needs only the invite link,
+`https://plugins.<base-domain>/usertest/?code=<USERTEST_CODE>`, and Chrome
+or Edge. The code is in `/etc/usertest-moderator.env` on the droplet;
+`moderator/run.sh` creates it on first run. The page:
+
+1. explains the session, what is recorded and where it goes, and asks for
+   consent;
+2. stores the test catalog URL in the browser, opens a fresh drive
+   (`/app/dev-drive`) in a second window, and starts recording the shared
+   screen and the microphone;
+3. listens with the browser's speech recognition (Chrome sends the audio to
+   Google), sends a turn to the moderator after a pause, a minute of silence
+   or a new error in the collector log, and speaks the answer with the
+   browser's speech synthesis.
+
+The moderator (`moderator/server.mjs`) asks Claude (`claude-opus-5`, effort
+`low`, server-side refusal fallback on) for the next line, following the
+interview script in `moderator/script.md`: short spoken questions, mostly
+listening (`[WAIT]`), no help unless the tester is stuck and asks. Each turn
+includes the collector's error, warning, feedback and sync lines since the
+previous turn. Measured on 2026-09-28: about 3 seconds per turn, and the
+script served from the prompt cache after the first turn.
+
+Per session, `/var/lib/usertest-sessions/<id>/` holds `meta.json`,
+`transcript.jsonl` (both sides, with the log lines each turn saw, and token
+usage) and `recording.webm`. Limits: the invite code on every request, at
+most 120 turns per session and 20 sessions per UTC day. The moderator keeps
+sessions in memory, so restarting it ends the sessions in progress.
+
+Not verified yet: a full session by a real tester, and Edge.
+
+## Without the moderator
 
 1. Open `https://plugins.<base-domain>/app/dev-drive`. It creates an agent
    and a drive in the browser, without signup.
 2. In Settings → Integration, set the plugin catalog URL to
    `https://catalog.<base-domain>/catalog.json`.
 3. On Integrations, install an app and connect it with your own account.
 
-Step 2 goes away once the planned `/usertest` page sets it.
-
 ## Logs
 
 The collector (`collector/server.mjs`, no dependencies, run in a
@@ -119,6 +152,10 @@ sends. Bodies over 1 MB are refused.
 Testers connect real accounts. Their data is on the droplet, in the Docker
 volume `atomic-plugins-store`, and their provider tokens are in
 localthought.io's database. Error reports can contain what was on screen
-(titles in messages, URLs), and they stay in `/var/lib/usertest-logs`. Tell testers this before they start. Reset the
+(titles in messages, URLs), and they stay in `/var/lib/usertest-logs`.
+Moderated sessions add screen and microphone recordings and transcripts in
+`/var/lib/usertest-sessions`; the transcripts also went through Google's
+speech recognition and Anthropic's API. Delete a session's folder after
+analysis. Tell testers this before they start. Reset the
 store with `docker rm -f atomic-plugins && docker volume rm atomic-plugins-store`,
 then run `server.sh` again.

usertest/catalog.mjs

@@ -43,7 +43,7 @@ const I = 'https://atomicdata.dev/integrations/properties/';
 
 /** Drive apps built here. `base` is the catalog entry whose copy they reuse. */
 const VERSIONS = {
-  calendar: 'usertest-3',
+  calendar: 'usertest-4',
   'issue-tracker': 'usertest',
   money: 'usertest',
   notion: 'usertest',

usertest/deploy.sh

@@ -19,7 +19,7 @@ HERE=$(cd "$(dirname "$0")" && pwd)
 # extended attributes that GNU tar on the droplet warns about.
 (cd "$HERE/out" && COPYFILE_DISABLE=1 tar --no-xattrs -czf - catalog.json apps) |
   ssh "$@" "$TARGET" 'mkdir -p /srv/catalog && tar xzf - -C /srv/catalog'
-(cd "$HERE" && COPYFILE_DISABLE=1 tar --no-xattrs -czf - Caddyfile caddy-usertest.conf server.sh collector) |
+(cd "$HERE" && COPYFILE_DISABLE=1 tar --no-xattrs --exclude node_modules -czf - Caddyfile caddy-usertest.conf server.sh collector moderator page) |
   ssh "$@" "$TARGET" 'set -e
     test -f /etc/caddy/usertest.env ||
       { echo "create /etc/caddy/usertest.env first (README.md)" >&2; exit 1; }

usertest/moderator/.gitignore

@@ -0,0 +1 @@
+node_modules/

usertest/moderator/package.json

@@ -0,0 +1,12 @@
+{
+  "name": "usertest-moderator",
+  "private": true,
+  "type": "module",
+  "description": "The voice moderator of user-testing sessions; see ../README.md",
+  "scripts": {
+    "start": "node server.mjs"
+  },
+  "dependencies": {
+    "@anthropic-ai/sdk": "0.128.0"
+  }
+}

usertest/moderator/run.sh

@@ -0,0 +1,21 @@
+#!/bin/sh
+# (Re)starts the voice moderator (server.mjs) on the droplet, as root:
+# `sh run.sh`. It reads the API key from /etc/anthropic.env and the invite
+# code from /etc/usertest-moderator.env (USERTEST_CODE=..., created on first
+# run). Sessions are kept in /var/lib/usertest-sessions/<id>/.
+set -eu
+HERE=$(cd "$(dirname "$0")" && pwd)
+test -s /etc/anthropic.env || { echo "put ANTHROPIC_API_KEY=... in /etc/anthropic.env" >&2; exit 1; }
+if [ ! -f /etc/usertest-moderator.env ]; then
+  (umask 077 && printf 'USERTEST_CODE=%s\n' "$(head -c 12 /dev/urandom | od -An -tx1 | tr -d ' \n')" > /etc/usertest-moderator.env)
+fi
+mkdir -p /var/lib/usertest-sessions /var/lib/usertest-logs
+docker run --rm -v "$HERE:/app" -w /app node:22-alpine npm ci --omit=dev --no-audit --no-fund --silent
+docker rm -f usertest-moderator >/dev/null 2>&1 || true
+docker run -d --name usertest-moderator --restart unless-stopped --init \
+  -p 127.0.0.1:8082:8082 \
+  --env-file /etc/anthropic.env --env-file /etc/usertest-moderator.env \
+  -v "$HERE:/app:ro" -w /app \
+  -v /var/lib/usertest-sessions:/sessions \
+  -v /var/lib/usertest-logs:/logs:ro \
+  node:22-alpine node server.mjs

usertest/moderator/script.md

@@ -0,0 +1,29 @@
+You are the moderator of a remote usability test of Atomic, a personal data app, and its new plugins ("drive apps") that import data from other services. You talk with one tester through their browser: everything you write is read aloud by a speech synthesizer, and what the tester says reaches you as speech-to-text, which can contain recognition errors. The tester has headphones on and is sharing their screen, which is recorded. With each turn you get a screenshot of the tester's shared screen, marked [Screen]; earlier turns show only that one was there. Use it to follow where they are, but don't describe it back to them and never tell them where to click. You also get the app's log lines (errors, syncs, feedback) since your last turn, in a block marked [Log].
+
+## Your goal
+
+Find out where people get stuck, what they expect, and what confuses them. The point is to observe, not to teach: a tester who is helped too early hides exactly the problems we want to find.
+
+## How to speak
+
+- One or two short sentences per turn, then stop. No lists, markdown, emoji, URLs or code: everything is spoken.
+- Speak English only, even if the tester answers in another language: the voice and the speech recognizer are set to English. If they speak another language, kindly ask them to continue in English.
+- Ask open questions: "What do you expect to happen?", "What are you looking for now?", "What do you make of that?" Never ask leading questions.
+- Encourage thinking aloud. If they fall silent (you get "(silence)"), gently ask what they are doing or thinking.
+- If they ask you to wait, say "Sure" once, then answer [WAIT] to every "(silence)" until they speak again.
+- Mostly listen. When the tester is just thinking aloud and making progress, answer with only the token [WAIT]: nothing is spoken, and you get their next words later. Speak when they ask you something, finish or give up a task, fall silent, or the [Log] shows an error.
+- Don't explain how the app works and don't say which button to press. Only when the tester has been stuck on the same step for several turns and asks for help, give the smallest possible hint, and note that you did.
+- If the [Log] shows an error, don't name it. Ask what they see on the screen and what they expected instead.
+- Stay neutral: don't praise or apologize for the app.
+
+## The session
+
+The tester already agreed to the recording before this started; don't ask again. Take roughly 20 to 30 minutes, one task at a time. Only move on when a task is done or the tester gives up.
+
+1. In your first turn, welcome them, ask them to think aloud, tell them a second window opened with their own empty Atomic drive and that we are testing the app, not them. End that same turn with task 2. Keep it to three short sentences.
+2. Task: "Bring the events from one of your calendars into Atomic." They can use their own Google account. If they prefer not to connect an account, let them skip to task 4.
+3. Task: "Change one of those events, and get the change back into your calendar." Afterwards ask whether they would trust this with their real calendar, and why.
+4. Task: "Look around what else Atomic can connect to, and try one that interests you."
+5. Wrap up: ask what was most confusing, what they liked, and whether they would use this and for what. Then thank them and say they can close the windows.
+
+After the thank-you, end your final message with the exact token [END] on its own. Never write [END] earlier.