From 3d70de33d9965d401aa4636386a34f3daa6ecfff Mon Sep 17 00:00:00 2001 From: Michiel de Jong Date: Mon, 28 Sep 2026 14:24:08 +0200 Subject: [PATCH 1/4] usertest: a voice-moderated session page and its Claude moderator Testers get one invite link to /usertest/ on the test instance. The page asks consent for what is recorded and where it goes, sets the test catalog, opens a fresh drive in a second window, records the shared screen and the microphone, listens with the browser's speech recognition and speaks the moderator's answers with speech synthesis. The moderator (moderator/server.mjs, @anthropic-ai/sdk, claude-opus-5 at effort low, server-side refusal fallback) follows moderator/script.md: short spoken questions, mostly listening ([WAIT]), no help unless the tester is stuck and asks. Every turn also carries the collector's error, warning, feedback and sync lines since the previous one. Per session it keeps the transcript and the recording. The endpoints need the invite code, and sessions and turns are capped, because they spend API money. Checked on the droplet: about 3 s per turn, [WAIT] while the tester thinks aloud, a question instead of a hint when asked which button to press, and prompt-cache reads from the second turn on. Not yet run by a real tester. Co-Authored-By: Claude Opus 5.5 --- usertest/Caddyfile | 16 +- usertest/README.md | 45 ++++- usertest/deploy.sh | 2 +- usertest/moderator/.gitignore | 1 + usertest/moderator/package-lock.json | 84 ++++++++ usertest/moderator/package.json | 12 ++ usertest/moderator/run.sh | 21 ++ usertest/moderator/script.md | 28 +++ usertest/moderator/server.mjs | 290 +++++++++++++++++++++++++++ usertest/page/app.js | 276 +++++++++++++++++++++++++ usertest/page/index.html | 155 ++++++++++++++ 11 files changed, 924 insertions(+), 6 deletions(-) create mode 100644 usertest/moderator/.gitignore create mode 100644 usertest/moderator/package-lock.json create mode 100644 usertest/moderator/package.json create mode 100755 usertest/moderator/run.sh create mode 100644 usertest/moderator/script.md create mode 100644 usertest/moderator/server.mjs create mode 100644 usertest/page/app.js create mode 100644 usertest/page/index.html diff --git a/usertest/Caddyfile b/usertest/Caddyfile index 1fa688ee..fb27a142 100644 --- a/usertest/Caddyfile +++ b/usertest/Caddyfile @@ -9,8 +9,22 @@ # atomic-server (server.sh), plain HTTP on loopback. It builds its links from # ATOMIC_DOMAIN and reads the scheme from X-Forwarded-Proto, which Caddy sets. +# /usertest/ is the moderated-session page (page/) and /usertest/api/ its +# moderator (moderator/). They share the data-browser's origin so the page can +# set the test catalog in the browser's storage. plugins.{$BASE_DOMAIN} { - reverse_proxy 127.0.0.1:8080 + redir /usertest /usertest/?{query} 308 + handle_path /usertest/api/* { + reverse_proxy 127.0.0.1:8082 + } + handle_path /usertest/* { + root * /opt/usertest/page + header Cache-Control "no-cache" + file_server + } + handle { + reverse_proxy 127.0.0.1:8080 + } } # The test catalog and the drive app modules it points at (catalog.mjs). diff --git a/usertest/README.md b/usertest/README.md index 6478b98a..0dbce40d 100644 --- a/usertest/README.md +++ b/usertest/README.md @@ -11,6 +11,7 @@ rebuild it. | `https://plugins.` | atomic-server, the published e2e image of the pinned commit (`server.sh`) | | `https://catalog./catalog.json` | the test catalog and the app modules it points at (`catalog.mjs`) | | `https://logs.` | the log collector (`collector/`) | +| `https://plugins./usertest/` | the moderated-session page (`page/`) and its voice moderator (`moderator/`, at `/usertest/api/`) | | `https://localthought.io` | the integration proxy, shared with everyone else (not on the droplet) | The base domain is currently `178-62-223-35.sslip.io`. sslip.io resolves @@ -60,6 +61,7 @@ for a in calendar money notion timesheets; do (cd integrations/$a && pnpm instal USERTEST_LOG_URL=https://logs.178-62-223-35.sslip.io/log node usertest/catalog.mjs sh usertest/deploy.sh root@178.62.223.35 ssh root@178.62.223.35 sh /opt/usertest/collector/run.sh +ssh root@178.62.223.35 sh /opt/usertest/moderator/run.sh # needs /etc/anthropic.env ssh root@178.62.223.35 sh /opt/usertest/server.sh 178-62-223-35.sslip.io ``` @@ -74,7 +76,40 @@ ssh root@178.62.223.35 sh /opt/usertest/server.sh 178-62-223-35.sslip.io Never rebuild into an existing version: the host refuses a module whose bytes no longer match the hash in the catalog. -## For testers +## Moderated sessions + +A tester needs only the invite link, +`https://plugins./usertest/?code=`, and Chrome +or Edge. The code is in `/etc/usertest-moderator.env` on the droplet; +`moderator/run.sh` creates it on first run. The page: + +1. explains the session, what is recorded and where it goes, and asks for + consent; +2. stores the test catalog URL in the browser, opens a fresh drive + (`/app/dev-drive`) in a second window, and starts recording the shared + screen and the microphone; +3. listens with the browser's speech recognition (Chrome sends the audio to + Google), sends a turn to the moderator after a pause, a minute of silence + or a new error in the collector log, and speaks the answer with the + browser's speech synthesis. + +The moderator (`moderator/server.mjs`) asks Claude (`claude-opus-5`, effort +`low`, server-side refusal fallback on) for the next line, following the +interview script in `moderator/script.md`: short spoken questions, mostly +listening (`[WAIT]`), no help unless the tester is stuck and asks. Each turn +includes the collector's error, warning, feedback and sync lines since the +previous turn. Measured on 2026-09-28: about 3 seconds per turn, and the +script served from the prompt cache after the first turn. + +Per session, `/var/lib/usertest-sessions//` holds `meta.json`, +`transcript.jsonl` (both sides, with the log lines each turn saw, and token +usage) and `recording.webm`. Limits: the invite code on every request, at +most 120 turns per session and 20 sessions per UTC day. The moderator keeps +sessions in memory, so restarting it ends the sessions in progress. + +Not verified yet: a full session by a real tester, and Edge. + +## Without the moderator 1. Open `https://plugins./app/dev-drive`. It creates an agent and a drive in the browser, without signup. @@ -82,8 +117,6 @@ no longer match the hash in the catalog. `https://catalog./catalog.json`. 3. On Integrations, install an app and connect it with your own account. -Step 2 goes away once the planned `/usertest` page sets it. - ## Logs The collector (`collector/server.mjs`, no dependencies, run in a @@ -119,6 +152,10 @@ sends. Bodies over 1 MB are refused. Testers connect real accounts. Their data is on the droplet, in the Docker volume `atomic-plugins-store`, and their provider tokens are in localthought.io's database. Error reports can contain what was on screen -(titles in messages, URLs), and they stay in `/var/lib/usertest-logs`. Tell testers this before they start. Reset the +(titles in messages, URLs), and they stay in `/var/lib/usertest-logs`. +Moderated sessions add screen and microphone recordings and transcripts in +`/var/lib/usertest-sessions`; the transcripts also went through Google's +speech recognition and Anthropic's API. Delete a session's folder after +analysis. Tell testers this before they start. Reset the store with `docker rm -f atomic-plugins && docker volume rm atomic-plugins-store`, then run `server.sh` again. diff --git a/usertest/deploy.sh b/usertest/deploy.sh index c92ded53..ac33f327 100755 --- a/usertest/deploy.sh +++ b/usertest/deploy.sh @@ -19,7 +19,7 @@ HERE=$(cd "$(dirname "$0")" && pwd) # extended attributes that GNU tar on the droplet warns about. (cd "$HERE/out" && COPYFILE_DISABLE=1 tar --no-xattrs -czf - catalog.json apps) | ssh "$@" "$TARGET" 'mkdir -p /srv/catalog && tar xzf - -C /srv/catalog' -(cd "$HERE" && COPYFILE_DISABLE=1 tar --no-xattrs -czf - Caddyfile caddy-usertest.conf server.sh collector) | +(cd "$HERE" && COPYFILE_DISABLE=1 tar --no-xattrs --exclude node_modules -czf - Caddyfile caddy-usertest.conf server.sh collector moderator page) | ssh "$@" "$TARGET" 'set -e test -f /etc/caddy/usertest.env || { echo "create /etc/caddy/usertest.env first (README.md)" >&2; exit 1; } diff --git a/usertest/moderator/.gitignore b/usertest/moderator/.gitignore new file mode 100644 index 00000000..c2658d7d --- /dev/null +++ b/usertest/moderator/.gitignore @@ -0,0 +1 @@ +node_modules/ diff --git a/usertest/moderator/package-lock.json b/usertest/moderator/package-lock.json new file mode 100644 index 00000000..3f8b61d6 --- /dev/null +++ b/usertest/moderator/package-lock.json @@ -0,0 +1,84 @@ +{ + "name": "usertest-moderator", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "usertest-moderator", + "dependencies": { + "@anthropic-ai/sdk": "0.128.0" + } + }, + "node_modules/@anthropic-ai/sdk": { + "version": "0.128.0", + "resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.128.0.tgz", + "integrity": "sha512-tl5cBFZC1jVFrTuQyvQObA4WmuNgcYVeXfriqOumgtLAHUm4gPUj18YnszBW60v0PrjDB8nBDOC6wd9CmHQFlA==", + "license": "MIT", + "dependencies": { + "json-schema-to-ts": "^3.1.1", + "standardwebhooks": "^1.0.0" + }, + "bin": { + "anthropic-ai-sdk": "bin/cli" + }, + "peerDependencies": { + "zod": "^3.25.0 || ^4.0.0" + }, + "peerDependenciesMeta": { + "zod": { + "optional": true + } + } + }, + "node_modules/@babel/runtime": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.7.tgz", + "integrity": "sha512-Nq8OhGWiZIZGV6hLHoyAKLLcJihP/xFeBMGJoUrxTX2psI8dCifzLhZISFb+VWS3wFMRDmCGw5R+dOySCqPLhw==", + "license": "MIT", + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@stablelib/base64": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/@stablelib/base64/-/base64-1.0.1.tgz", + "integrity": "sha512-1bnPQqSxSuc3Ii6MhBysoWCg58j97aUjuCSZrGSmDxNqtytIi0k8utUenAwTZN4V5mXXYGsVUI9zeBqy+jBOSQ==", + "license": "MIT" + }, + "node_modules/fast-sha256": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/fast-sha256/-/fast-sha256-1.3.0.tgz", + "integrity": "sha512-n11RGP/lrWEFI/bWdygLxhI+pVeo1ZYIVwvvPkW7azl/rOy+F3HYRZ2K5zeE9mmkhQppyv9sQFx0JM9UabnpPQ==", + "license": "Unlicense" + }, + "node_modules/json-schema-to-ts": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/json-schema-to-ts/-/json-schema-to-ts-3.1.1.tgz", + "integrity": "sha512-+DWg8jCJG2TEnpy7kOm/7/AxaYoaRbjVB4LFZLySZlWn8exGs3A4OLJR966cVvU26N7X9TWxl+Jsw7dzAqKT6g==", + "license": "MIT", + "dependencies": { + "@babel/runtime": "^7.18.3", + "ts-algebra": "^2.0.0" + }, + "engines": { + "node": ">=16" + } + }, + "node_modules/standardwebhooks": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/standardwebhooks/-/standardwebhooks-1.1.1.tgz", + "integrity": "sha512-bCbX9ZEyFkWPsRz7Bl3NuQUJohmwGSev/yhr7vhaGPlc4AfIrspIRa6cPTBuI1ItmrTDJ4d/S2hCsfe4+vQGnQ==", + "license": "MIT", + "dependencies": { + "@stablelib/base64": "^1.0.0", + "fast-sha256": "^1.3.0" + } + }, + "node_modules/ts-algebra": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/ts-algebra/-/ts-algebra-2.0.0.tgz", + "integrity": "sha512-FPAhNPFMrkwz76P7cdjdmiShwMynZYN6SgOujD1urY4oNm80Ou9oMdmbR45LotcKOXoy7wSmHkRFE6Mxbrhefw==", + "license": "MIT" + } + } +} diff --git a/usertest/moderator/package.json b/usertest/moderator/package.json new file mode 100644 index 00000000..a7019634 --- /dev/null +++ b/usertest/moderator/package.json @@ -0,0 +1,12 @@ +{ + "name": "usertest-moderator", + "private": true, + "type": "module", + "description": "The voice moderator of user-testing sessions; see ../README.md", + "scripts": { + "start": "node server.mjs" + }, + "dependencies": { + "@anthropic-ai/sdk": "0.128.0" + } +} diff --git a/usertest/moderator/run.sh b/usertest/moderator/run.sh new file mode 100755 index 00000000..47089e7a --- /dev/null +++ b/usertest/moderator/run.sh @@ -0,0 +1,21 @@ +#!/bin/sh +# (Re)starts the voice moderator (server.mjs) on the droplet, as root: +# `sh run.sh`. It reads the API key from /etc/anthropic.env and the invite +# code from /etc/usertest-moderator.env (USERTEST_CODE=..., created on first +# run). Sessions are kept in /var/lib/usertest-sessions//. +set -eu +HERE=$(cd "$(dirname "$0")" && pwd) +test -s /etc/anthropic.env || { echo "put ANTHROPIC_API_KEY=... in /etc/anthropic.env" >&2; exit 1; } +if [ ! -f /etc/usertest-moderator.env ]; then + (umask 077 && printf 'USERTEST_CODE=%s\n' "$(head -c 12 /dev/urandom | od -An -tx1 | tr -d ' \n')" > /etc/usertest-moderator.env) +fi +mkdir -p /var/lib/usertest-sessions /var/lib/usertest-logs +docker run --rm -v "$HERE:/app" -w /app node:22-alpine npm ci --omit=dev --no-audit --no-fund --silent +docker rm -f usertest-moderator >/dev/null 2>&1 || true +docker run -d --name usertest-moderator --restart unless-stopped --init \ + -p 127.0.0.1:8082:8082 \ + --env-file /etc/anthropic.env --env-file /etc/usertest-moderator.env \ + -v "$HERE:/app:ro" -w /app \ + -v /var/lib/usertest-sessions:/sessions \ + -v /var/lib/usertest-logs:/logs:ro \ + node:22-alpine node server.mjs diff --git a/usertest/moderator/script.md b/usertest/moderator/script.md new file mode 100644 index 00000000..ee073678 --- /dev/null +++ b/usertest/moderator/script.md @@ -0,0 +1,28 @@ +You are the moderator of a remote usability test of Atomic, a personal data app, and its new plugins ("drive apps") that import data from other services. You talk with one tester through their browser: everything you write is read aloud by a speech synthesizer, and what the tester says reaches you as speech-to-text, which can contain recognition errors. The tester has headphones on and is sharing their screen, which is recorded. You cannot see the screen. You do get the app's log lines (errors, syncs, feedback) since your last turn, in a block marked [Log]. + +## Your goal + +Find out where people get stuck, what they expect, and what confuses them. The point is to observe, not to teach: a tester who is helped too early hides exactly the problems we want to find. + +## How to speak + +- One or two short sentences per turn, then stop. No lists, markdown, emoji, URLs or code: everything is spoken. +- Speak the tester's language. Start in English; switch if they answer in another language, for example Dutch. +- Ask open questions: "What do you expect to happen?", "What are you looking for now?", "What do you make of that?" Never ask leading questions. +- Encourage thinking aloud. If they fall silent (you get "(silence)"), gently ask what they are doing or thinking. +- Mostly listen. When the tester is just thinking aloud and making progress, answer with only the token [WAIT]: nothing is spoken, and you get their next words later. Speak when they ask you something, finish or give up a task, fall silent, or the [Log] shows an error. +- Don't explain how the app works and don't say which button to press. Only when the tester has been stuck on the same step for several turns and asks for help, give the smallest possible hint, and note that you did. +- If the [Log] shows an error, don't name it. Ask what they see on the screen and what they expected instead. +- Stay neutral: don't praise or apologize for the app. + +## The session + +The tester already agreed to the recording before this started; don't ask again. Take roughly 20 to 30 minutes, one task at a time. Only move on when a task is done or the tester gives up. + +1. In your first turn, welcome them, ask them to think aloud, tell them a second window opened with their own empty Atomic drive and that we are testing the app, not them. End that same turn with task 2. Keep it to three short sentences. +2. Task: "Bring the events from one of your calendars into Atomic." They can use their own Google account. If they prefer not to connect an account, let them skip to task 4. +3. Task: "Change one of those events, and get the change back into your calendar." Afterwards ask whether they would trust this with their real calendar, and why. +4. Task: "Look around what else Atomic can connect to, and try one that interests you." +5. Wrap up: ask what was most confusing, what they liked, and whether they would use this and for what. Then thank them and say they can close the windows. + +After the thank-you, end your final message with the exact token [END] on its own. Never write [END] earlier. diff --git a/usertest/moderator/server.mjs b/usertest/moderator/server.mjs new file mode 100644 index 00000000..9d1bde72 --- /dev/null +++ b/usertest/moderator/server.mjs @@ -0,0 +1,290 @@ +#!/usr/bin/env node +/** + * The voice moderator of a user-testing session (../page/ is its front end). + * Each turn gets what the tester said (speech-to-text in their browser) plus + * the collector's log lines since the previous turn, asks Claude for the next + * thing to say (script.md is the interview script), and returns it for the + * browser to speak. + * + * Per session it keeps, under SESSIONS_DIR//: meta.json, transcript.jsonl + * (both sides, with the log lines each turn saw) and recording.webm (screen + * and microphone, uploaded by the page in chunks). + * + * The endpoints spend API money and are public, so every request needs the + * invite code (USERTEST_CODE, sent as `x-usertest-code`), a session takes at + * most MAX_TURNS turns, and at most MAX_SESSIONS_PER_DAY sessions start per + * UTC day. ANTHROPIC_API_KEY comes from the environment (/etc/anthropic.env + * on the droplet, passed by run.sh) and never leaves this process. + */ +import Anthropic from '@anthropic-ai/sdk'; +import { randomBytes } from 'node:crypto'; +import { + appendFileSync, + existsSync, + mkdirSync, + readFileSync, + readdirSync, + writeFileSync, +} from 'node:fs'; +import { createServer } from 'node:http'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const here = dirname(fileURLToPath(import.meta.url)); +const PORT = Number(process.env.PORT ?? 8082); +const SESSIONS_DIR = process.env.SESSIONS_DIR ?? '/sessions'; +const LOG_DIR = process.env.LOG_DIR ?? '/logs'; +const CODE = process.env.USERTEST_CODE; +const MODEL = process.env.MODERATOR_MODEL ?? 'claude-opus-5'; +const MAX_TURNS = 120; +const MAX_SESSIONS_PER_DAY = 20; +const MAX_CHUNK = 32 * 1024 * 1024; +const MAX_LOG_LINES = 15; + +if (!CODE || CODE.length < 12) + throw new Error('USERTEST_CODE must be set (at least 12 characters)'); + +const SCRIPT = readFileSync(join(here, 'script.md'), 'utf8'); +const client = new Anthropic(); +/** id -> { dir, started, cursor, messages, turns, done } */ +const sessions = new Map(); + +mkdirSync(SESSIONS_DIR, { recursive: true }); + +const today = () => new Date().toISOString().slice(0, 10); + +function sessionsToday() { + return readdirSync(SESSIONS_DIR).filter(name => name.startsWith(today())) + .length; +} + +/** The collector's lines after `cursor` (an ISO time), made short enough to + * read: errors, warnings, feedback and sync outcomes only. */ +function logSince(cursor) { + const days = [...new Set([cursor.slice(0, 10), today()])]; + const lines = []; + + for (const day of days) { + const file = join(LOG_DIR, `${day}.jsonl`); + if (!existsSync(file)) continue; + + for (const raw of readFileSync(file, 'utf8').split('\n')) { + if (!raw) continue; + let entry; + + try { + entry = JSON.parse(raw); + } catch { + continue; + } + + if (entry.t <= cursor) continue; + const data = entry.data ?? {}; + const level = entry.level ?? data.level; + const message = + entry.type === 'feedback' + ? `feedback: ${entry.feedback?.message ?? ''}` + : (data.message ?? + entry.message ?? + entry.exceptions?.map(e => `${e.type}: ${e.value}`).join('; ')); + if (!message || !['error', 'warn', 'info', undefined].includes(level)) + continue; + if (level === 'info' && data.message !== 'Sync finished') continue; + const detail = + data.message === 'Sync finished' + ? ` (imported ${data.total}, unreadable ${data.skipped?.unreadable ?? 0})` + : ''; + lines.push( + `${entry.t.slice(11, 19)} ${entry.source} ${level ?? ''}: ${String(message).slice(0, 300)}${detail}`, + ); + } + } + + return lines.slice(-MAX_LOG_LINES); +} + +function record(session, entry) { + appendFileSync( + join(session.dir, 'transcript.jsonl'), + JSON.stringify({ t: new Date().toISOString(), ...entry }) + '\n', + ); +} + +/** Asks Claude for the next thing to say. */ +async function nextLine(session, said) { + const now = new Date().toISOString(); + const log = logSince(session.cursor); + session.cursor = now; + const heard = said.trim() || '(silence)'; + const content = [ + log.length ? `[Log]\n${log.join('\n')}` : '[Log]\n(nothing new)', + `[Tester]\n${session.turns === 0 ? '(the session starts now)' : heard}`, + ].join('\n\n'); + + session.messages.push({ role: 'user', content }); + record(session, { role: 'tester', said: heard, log }); + + const response = await client.beta.messages.create({ + model: MODEL, + max_tokens: 2000, + betas: ['server-side-fallback-2026-07-01'], + fallbacks: 'default', + // Spoken turns are short; low effort keeps the pause before each reply + // short too. + output_config: { effort: 'low' }, + cache_control: { type: 'ephemeral' }, + system: SCRIPT, + messages: session.messages, + }); + + const text = + response.stop_reason === 'refusal' + ? '' + : response.content + .filter(block => block.type === 'text') + .map(block => block.text) + .join(' ') + .trim(); + const done = text.includes('[END]'); + const wait = !done && /^\[WAIT\]$/.test(text); + const say = wait + ? '' + : text.replace('[END]', '').replace('[WAIT]', '').trim() || + 'Sorry, I lost my train of thought. Could you tell me what you are doing now?'; + + session.messages.push({ role: 'assistant', content: wait ? '[WAIT]' : say }); + session.turns++; + session.done = done; + record(session, { + role: 'moderator', + say, + done, + stop: response.stop_reason, + usage: response.usage, + }); + + return { say, done }; +} + +function readBody(req, limit) { + return new Promise((resolve, reject) => { + const chunks = []; + let size = 0; + req.on('data', chunk => { + size += chunk.length; + if (size > limit) { + reject(Object.assign(new Error('too large'), { status: 413 })); + req.destroy(); + } else chunks.push(chunk); + }); + req.on('end', () => resolve(Buffer.concat(chunks))); + req.on('error', reject); + }); +} + +createServer(async (req, res) => { + const url = new URL(req.url ?? '/', 'http://moderator'); + + const reply = (status, body) => { + res.writeHead(status, { 'content-type': 'application/json' }); + res.end(JSON.stringify(body ?? {})); + }; + + if (req.method === 'GET' && url.pathname === '/health') + return reply(200, { ok: true }); + if (req.headers['x-usertest-code'] !== CODE) + return reply(403, { error: 'Unknown invite code' }); + + try { + if (req.method === 'POST' && url.pathname === '/sessions') { + if (sessionsToday() >= MAX_SESSIONS_PER_DAY) + return reply(429, { error: 'No more sessions today' }); + const body = JSON.parse((await readBody(req, 64 * 1024)).toString()); + const id = `${today()}-${randomBytes(4).toString('hex')}`; + const dir = join(SESSIONS_DIR, id); + mkdirSync(dir); + const started = new Date().toISOString(); + writeFileSync( + join(dir, 'meta.json'), + JSON.stringify( + { + id, + started, + name: String(body.name ?? '').slice(0, 80), + lang: String(body.lang ?? '').slice(0, 20), + userAgent: req.headers['user-agent'], + model: MODEL, + }, + null, + 2, + ), + ); + sessions.set(id, { + dir, + started, + cursor: started, + messages: [], + turns: 0, + done: false, + }); + + return reply(200, { id }); + } + + const match = url.pathname.match( + /^\/sessions\/([\w-]+)\/(turn|recording|end|news)$/, + ); + const session = match && sessions.get(match[1]); + if (!session) return reply(404, { error: 'Unknown session' }); + + // Errors logged since the last turn, so the page can let the moderator + // react to them without waiting for the tester to speak. + if (req.method === 'GET' && match[2] === 'news') + return reply(200, { + errors: logSince(session.cursor).filter(line => / error: /.test(line)) + .length, + }); + + if (req.method === 'POST' && match[2] === 'turn') { + if (session.done || session.turns >= MAX_TURNS) + return reply(200, { say: '', done: true }); + const body = JSON.parse((await readBody(req, 64 * 1024)).toString()); + + return reply(200, await nextLine(session, String(body.said ?? ''))); + } + + if (req.method === 'POST' && match[2] === 'recording') { + appendFileSync( + join(session.dir, 'recording.webm'), + await readBody(req, MAX_CHUNK), + ); + + return reply(204); + } + + if (req.method === 'POST' && match[2] === 'end') { + record(session, { role: 'page', event: 'end' }); + session.done = true; + + return reply(200, { ok: true }); + } + + return reply(404, { error: 'Not found' }); + } catch (error) { + const status = + error.status === 413 + ? 413 + : error instanceof Anthropic.RateLimitError + ? 503 + : error instanceof Anthropic.APIError + ? 502 + : 500; + process.stderr.write( + `${new Date().toISOString()} ${error.stack ?? error}\n`, + ); + + return reply(status, { error: 'The moderator could not answer' }); + } +}).listen(PORT, () => { + process.stdout.write(`moderator on :${PORT} using ${MODEL}\n`); +}); diff --git a/usertest/page/app.js b/usertest/page/app.js new file mode 100644 index 00000000..3b6b146a --- /dev/null +++ b/usertest/page/app.js @@ -0,0 +1,276 @@ +// The tester's side of a moderated session: records screen and microphone, +// turns speech into text with the browser's recognizer, sends turns to the +// moderator (../moderator/server.mjs) and speaks its answers. Served at +// /usertest/ on the test instance, so it shares the data-browser's origin +// and can set the test catalog before opening the app. + +const API = '/usertest/api'; +const CATALOG_URL = `https://catalog.${location.hostname.replace(/^plugins\./, '')}/catalog.json`; +/** Send a turn this long after the tester stops talking (a question sooner). */ +const PAUSE_AFTER_QUESTION = 3000; +const PAUSE_AFTER_SPEECH = 9000; +/** Ask what's happening after this much silence. */ +const SILENCE = 60000; + +const $ = id => document.getElementById(id); +const Recognition = window.SpeechRecognition ?? window.webkitSpeechRecognition; +const params = new URLSearchParams(location.search); + +function inviteCode() { + const fromUrl = params.get('code'); + if (fromUrl) sessionStorage.setItem('usertest-code', fromUrl); + + return fromUrl ?? sessionStorage.getItem('usertest-code') ?? ''; +} + +const code = inviteCode(); +let session; +let recognition; +let recorder; +let speaking = false; +let ended = false; +let busy = false; +let heard = []; +let lastSpeech = Date.now(); +let lastTurn = Date.now(); +let uploads = Promise.resolve(); + +async function api(path, init = {}) { + const response = await fetch(`${API}${path}`, { + ...init, + headers: { 'x-usertest-code': code, ...(init.headers ?? {}) }, + }); + if (!response.ok) throw new Error(`${path}: ${response.status}`); + + return response.status === 204 ? {} : response.json(); +} + +function show(who, text) { + const p = document.createElement('p'); + if (who === 'me') p.className = 'me'; + p.textContent = who === 'me' ? `You: ${text}` : text; + $('log').prepend(p); +} + +function status(text) { + $('status').textContent = text; +} + +function speak(text) { + return new Promise(resolve => { + const utterance = new SpeechSynthesisUtterance(text); + const lang = $('lang').value; + utterance.lang = lang; + const voice = speechSynthesis + .getVoices() + .find(v => v.lang.replace('_', '-').startsWith(lang.slice(0, 2))); + if (voice) utterance.voice = voice; + speaking = true; + // The recognizer would otherwise hear the moderator (without headphones). + recognition?.abort(); + + utterance.onend = utterance.onerror = () => { + speaking = false; + listen(); + resolve(); + }; + + speechSynthesis.speak(utterance); + }); +} + +async function turn(said) { + if (busy || ended) return; + busy = true; + heard = []; + lastTurn = Date.now(); + if (said) show('me', said); + status('Thinking…'); + + try { + const { say, done } = await api(`/sessions/${session}/turn`, { + method: 'POST', + body: JSON.stringify({ said }), + }); + + if (say) { + show('moderator', say); + status('Speaking…'); + await speak(say); + } + + if (done) return finish(); + status('Listening…'); + } catch (error) { + status('The moderator is not answering. Keep going; it will try again.'); + console.error(error); + } finally { + busy = false; + } +} + +function listen() { + if (ended || speaking || !recognition) return; + + try { + recognition.start(); + } catch { + // Already running. + } +} + +function setupRecognition() { + recognition = new Recognition(); + recognition.lang = $('lang').value; + recognition.continuous = true; + recognition.interimResults = true; + + recognition.onresult = event => { + let interim = ''; + + for (let i = event.resultIndex; i < event.results.length; i++) { + const text = event.results[i][0].transcript.trim(); + if (event.results[i].isFinal) heard.push(text); + else interim += text; + } + + lastSpeech = Date.now(); + $('heard').textContent = [...heard, interim].join(' '); + }; + + // Chrome stops after a while of silence; keep listening until the end. + recognition.onend = () => listen(); + + recognition.onerror = event => { + if (event.error === 'not-allowed') + status('Microphone access is blocked for this page.'); + }; + + listen(); +} + +/** Decides when to hand the conversation to the moderator. */ +function tick() { + if (ended || busy || speaking) return; + const now = Date.now(); + const text = heard.join(' ').trim(); + + if (text) { + const wait = /\?\s*$/.test(text) + ? PAUSE_AFTER_QUESTION + : PAUSE_AFTER_SPEECH; + if (now - lastSpeech > wait) turn(text); + } else if (now - Math.max(lastSpeech, lastTurn) > SILENCE) turn(''); +} + +async function checkNews() { + if (ended || busy || speaking) return; + + try { + const { errors } = await api(`/sessions/${session}/news`); + if (errors && Date.now() - lastSpeech > 4000) turn(heard.join(' ').trim()); + } catch { + // Next time. + } +} + +async function startRecording() { + const screen = await navigator.mediaDevices.getDisplayMedia({ + video: { frameRate: 5 }, + audio: false, + }); + const mic = await navigator.mediaDevices.getUserMedia({ audio: true }); + const stream = new MediaStream([ + ...screen.getVideoTracks(), + ...mic.getAudioTracks(), + ]); + const type = ['video/webm;codecs=vp9,opus', 'video/webm'].find(t => + MediaRecorder.isTypeSupported(t), + ); + recorder = new MediaRecorder(stream, type ? { mimeType: type } : {}); + + recorder.ondataavailable = event => { + if (!event.data.size) return; + // In order, one chunk at a time: the server appends them to one file. + uploads = uploads.then(() => + api(`/sessions/${session}/recording`, { + method: 'POST', + body: event.data, + }).catch(error => console.error(error)), + ); + }; + + recorder.start(10000); + $('rec').textContent = '● recording'; + // Stopping the share from the browser's bar ends the session. + screen.getVideoTracks()[0].addEventListener('ended', () => finish()); +} + +async function finish() { + if (ended) return; + ended = true; + recognition?.abort(); + speechSynthesis.cancel(); + if (recorder?.state === 'recording') recorder.stop(); + recorder?.stream.getTracks().forEach(track => track.stop()); + status('Saving the recording…'); + // Let the last chunk arrive before saying goodbye. + await new Promise(resolve => setTimeout(resolve, 500)); + await uploads; + await api(`/sessions/${session}/end`, { method: 'POST' }).catch(() => {}); + $('rec').textContent = ''; + status('Thank you! The session has ended; you can close both windows.'); + $('end').hidden = true; +} + +async function start() { + $('error').hidden = true; + $('start').disabled = true; + // The data-browser reads the catalog URL from this origin's storage. + localStorage.setItem('plugin-catalog-url', CATALOG_URL); + const width = Math.round(screen.availWidth * 0.62); + const app = window.open( + '/app/dev-drive', + 'atomic-usertest-app', + `popup,width=${width},height=${screen.availHeight},left=${screen.availWidth - width},top=0`, + ); + + try { + ({ id: session } = await api('/sessions', { + method: 'POST', + body: JSON.stringify({ name: $('name').value, lang: $('lang').value }), + })); + await startRecording(); + } catch (error) { + app?.close(); + $('error').textContent = + error.name === 'NotAllowedError' + ? 'The session needs screen sharing and the microphone. Please allow both and try again.' + : `Could not start the session (${error.message}). Is the invite link complete?`; + $('error').hidden = false; + $('start').disabled = false; + + return; + } + + $('intro').hidden = true; + $('consent').hidden = true; + $('session').hidden = false; + setupRecognition(); + setInterval(tick, 1000); + setInterval(checkNews, 5000); + turn(''); +} + +if (!Recognition) $('unsupported').hidden = false; + +if (!code) { + $('error').textContent = 'This page needs the invite link you received.'; + $('error').hidden = false; +} + +$('agree').addEventListener('change', () => { + $('start').disabled = !$('agree').checked || !Recognition || !code; +}); +$('start').addEventListener('click', start); +$('end').addEventListener('click', () => finish()); diff --git a/usertest/page/index.html b/usertest/page/index.html new file mode 100644 index 00000000..54238486 --- /dev/null +++ b/usertest/page/index.html @@ -0,0 +1,155 @@ + + + + + + Atomic user test + + + +
+

Try Atomic's new plugins

+ +
+

+ Thanks for helping! For about 25 minutes you'll try a new part of + Atomic while a voice moderator (Claude, an AI) asks you what you + see and think. There are no wrong answers: we are testing the app, + not you. +

+
    +
  • Find a quiet place where you can talk, and put on headphones.
    • +
  • Use Chrome or Edge on a laptop or desktop.
    • +
  • + When you start, a second window opens with your own empty Atomic + drive. Put it next to this one; keep this window open. +
    • +
+ +
+ + + + +
+ + + From 68b967f6927960e4861f0b05d94500286a6cb8a1 Mon Sep 17 00:00:00 2001 From: Michiel de Jong Date: Mon, 28 Sep 2026 14:25:58 +0200 Subject: [PATCH 2/4] usertest: calendar usertest-4 (reads all-day events with end == start, #184) Co-Authored-By: Claude Opus 5.5 --- usertest/catalog.mjs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/usertest/catalog.mjs b/usertest/catalog.mjs index ca146946..5045c81c 100755 --- a/usertest/catalog.mjs +++ b/usertest/catalog.mjs @@ -43,7 +43,7 @@ const I = 'https://atomicdata.dev/integrations/properties/'; /** Drive apps built here. `base` is the catalog entry whose copy they reuse. */ const VERSIONS = { - calendar: 'usertest-3', + calendar: 'usertest-4', 'issue-tracker': 'usertest', money: 'usertest', notion: 'usertest', From 2f5f59bd0aeb96659b4dc47a00ff5ece9536f755 Mon Sep 17 00:00:00 2001 From: Michiel de Jong Date: Mon, 28 Sep 2026 15:20:48 +0200 Subject: [PATCH 3/4] usertest: English only, one voice, questions by their words, screenshots From the first real session (2026-09-28, Michiel): - Dutch was chosen at the start, but the moderator switched language on its own while the voice stayed; and before Chrome's voice list loaded, the first line had a different voice. Now English only (page, recognizer, script) and one voice picked once the list has loaded. - Questions did not hand over: Chrome's recognizer adds no "?", so every question waited the full 9 s. Questions are now recognized by their words (2 s), other speech hands over after 5 s, and an "Ask the moderator" button hands over at once. - The moderator could not see the screen. Each turn now carries a 1280-pixel JPEG of the shared screen (the history keeps only the text), saved per turn as screen-NNN.jpg; the consent text says so. Checked on the droplet: a turn with a JPEG is accepted and answered. Co-Authored-By: Claude Opus 5.5 --- usertest/moderator/script.md | 4 +- usertest/moderator/server.mjs | 50 ++++++++++++++++--- usertest/page/app.js | 91 ++++++++++++++++++++++++++++++----- usertest/page/index.html | 15 +++--- 4 files changed, 132 insertions(+), 28 deletions(-) diff --git a/usertest/moderator/script.md b/usertest/moderator/script.md index ee073678..88c126f7 100644 --- a/usertest/moderator/script.md +++ b/usertest/moderator/script.md @@ -1,4 +1,4 @@ -You are the moderator of a remote usability test of Atomic, a personal data app, and its new plugins ("drive apps") that import data from other services. You talk with one tester through their browser: everything you write is read aloud by a speech synthesizer, and what the tester says reaches you as speech-to-text, which can contain recognition errors. The tester has headphones on and is sharing their screen, which is recorded. You cannot see the screen. You do get the app's log lines (errors, syncs, feedback) since your last turn, in a block marked [Log]. +You are the moderator of a remote usability test of Atomic, a personal data app, and its new plugins ("drive apps") that import data from other services. You talk with one tester through their browser: everything you write is read aloud by a speech synthesizer, and what the tester says reaches you as speech-to-text, which can contain recognition errors. The tester has headphones on and is sharing their screen, which is recorded. With each turn you get a screenshot of the tester's shared screen, marked [Screen]; earlier turns show only that one was there. Use it to follow where they are, but don't describe it back to them and never tell them where to click. You also get the app's log lines (errors, syncs, feedback) since your last turn, in a block marked [Log]. ## Your goal @@ -7,7 +7,7 @@ Find out where people get stuck, what they expect, and what confuses them. The p ## How to speak - One or two short sentences per turn, then stop. No lists, markdown, emoji, URLs or code: everything is spoken. -- Speak the tester's language. Start in English; switch if they answer in another language, for example Dutch. +- Speak English only, even if the tester answers in another language: the voice and the speech recognizer are set to English. If they speak another language, kindly ask them to continue in English. - Ask open questions: "What do you expect to happen?", "What are you looking for now?", "What do you make of that?" Never ask leading questions. - Encourage thinking aloud. If they fall silent (you get "(silence)"), gently ask what they are doing or thinking. - Mostly listen. When the tester is just thinking aloud and making progress, answer with only the token [WAIT]: nothing is spoken, and you get their next words later. Speak when they ask you something, finish or give up a task, fall silent, or the [Log] shows an error. diff --git a/usertest/moderator/server.mjs b/usertest/moderator/server.mjs index 9d1bde72..c7ea4967 100644 --- a/usertest/moderator/server.mjs +++ b/usertest/moderator/server.mjs @@ -111,18 +111,50 @@ function record(session, entry) { } /** Asks Claude for the next thing to say. */ -async function nextLine(session, said) { +async function nextLine(session, said, screenshot) { const now = new Date().toISOString(); const log = logSince(session.cursor); session.cursor = now; const heard = said.trim() || '(silence)'; + const shot = + typeof screenshot === 'string' && /^[A-Za-z0-9+/=]+$/.test(screenshot) + ? screenshot + : undefined; const content = [ log.length ? `[Log]\n${log.join('\n')}` : '[Log]\n(nothing new)', `[Tester]\n${session.turns === 0 ? '(the session starts now)' : heard}`, ].join('\n\n'); - session.messages.push({ role: 'user', content }); - record(session, { role: 'tester', said: heard, log }); + // History keeps the text only; the screenshot goes with this turn alone, + // so the conversation does not grow by an image per turn. + session.messages.push({ + role: 'user', + content: shot + ? `[Screen] (screenshot shown at the time)\n\n${content}` + : content, + }); + if (shot) + writeFileSync( + join(session.dir, `screen-${String(session.turns).padStart(3, '0')}.jpg`), + Buffer.from(shot, 'base64'), + ); + record(session, { role: 'tester', said: heard, log, screenshot: !!shot }); + + const messages = shot + ? [ + ...session.messages.slice(0, -1), + { + role: 'user', + content: [ + { + type: 'image', + source: { type: 'base64', media_type: 'image/jpeg', data: shot }, + }, + { type: 'text', text: `[Screen] (the image above)\n\n${content}` }, + ], + }, + ] + : session.messages; const response = await client.beta.messages.create({ model: MODEL, @@ -134,7 +166,7 @@ async function nextLine(session, said) { output_config: { effort: 'low' }, cache_control: { type: 'ephemeral' }, system: SCRIPT, - messages: session.messages, + messages, }); const text = @@ -248,9 +280,15 @@ createServer(async (req, res) => { if (req.method === 'POST' && match[2] === 'turn') { if (session.done || session.turns >= MAX_TURNS) return reply(200, { say: '', done: true }); - const body = JSON.parse((await readBody(req, 64 * 1024)).toString()); + // Room for a 1280-pixel JPEG screenshot, base64-encoded. + const body = JSON.parse( + (await readBody(req, 4 * 1024 * 1024)).toString(), + ); - return reply(200, await nextLine(session, String(body.said ?? ''))); + return reply( + 200, + await nextLine(session, String(body.said ?? ''), body.screenshot), + ); } if (req.method === 'POST' && match[2] === 'recording') { diff --git a/usertest/page/app.js b/usertest/page/app.js index 3b6b146a..6871bd1d 100644 --- a/usertest/page/app.js +++ b/usertest/page/app.js @@ -6,9 +6,15 @@ const API = '/usertest/api'; const CATALOG_URL = `https://catalog.${location.hostname.replace(/^plugins\./, '')}/catalog.json`; -/** Send a turn this long after the tester stops talking (a question sooner). */ -const PAUSE_AFTER_QUESTION = 3000; -const PAUSE_AFTER_SPEECH = 9000; +/** Send a turn this long after the tester stops talking (a question sooner). + * Chrome's recognizer adds no punctuation, so questions are recognized by + * their words; the moderator answers [WAIT] when nothing needs saying. */ +const PAUSE_AFTER_QUESTION = 2000; +const PAUSE_AFTER_SPEECH = 5000; +const QUESTION = + /\b(how|what|where|why|which|who|when|can you|could you|should i|do i|is it|is there|are there)\b/i; +/** Screenshots sent with a turn: at most this wide, as JPEG. */ +const SHOT_WIDTH = 1280; /** Ask what's happening after this much silence. */ const SILENCE = 60000; @@ -34,6 +40,8 @@ let heard = []; let lastSpeech = Date.now(); let lastTurn = Date.now(); let uploads = Promise.resolve(); +/** Plays the shared screen off-screen, so a turn can grab a frame of it. */ +let screenVideo; async function api(path, init = {}) { const response = await fetch(`${API}${path}`, { @@ -56,14 +64,49 @@ function status(text) { $('status').textContent = text; } -function speak(text) { +/** English only for now: in a Dutch trial (2026-09-28) the moderator switched + * language on its own while the voice stayed on the one picked at the start. */ +const LANG = 'en-US'; + +/** One voice for the whole session. Chrome fills getVoices() only after + * `voiceschanged`, so without waiting the first line got the default voice + * and later lines another one. */ +let voice; + +function pickVoice() { + const english = speechSynthesis + .getVoices() + .filter(v => v.lang.replace('_', '-') === LANG); + + return ( + english.find(v => /google/i.test(v.name)) ?? + english.find(v => v.localService) ?? + english[0] + ); +} + +function voiceReady() { + voice ??= pickVoice(); + if (voice) return Promise.resolve(); + + return new Promise(resolve => { + const done = () => { + voice ??= pickVoice(); + resolve(); + }; + + speechSynthesis.addEventListener('voiceschanged', done, { once: true }); + // Some browsers never fire it; go on with the default voice. + setTimeout(done, 2000); + }); +} + +async function speak(text) { + await voiceReady(); + return new Promise(resolve => { const utterance = new SpeechSynthesisUtterance(text); - const lang = $('lang').value; - utterance.lang = lang; - const voice = speechSynthesis - .getVoices() - .find(v => v.lang.replace('_', '-').startsWith(lang.slice(0, 2))); + utterance.lang = LANG; if (voice) utterance.voice = voice; speaking = true; // The recognizer would otherwise hear the moderator (without headphones). @@ -79,10 +122,24 @@ function speak(text) { }); } +/** The shared screen as it is now: base64 JPEG, or undefined. */ +function screenshot() { + const video = screenVideo; + if (!video?.videoWidth) return undefined; + const scale = Math.min(1, SHOT_WIDTH / video.videoWidth); + const canvas = document.createElement('canvas'); + canvas.width = Math.round(video.videoWidth * scale); + canvas.height = Math.round(video.videoHeight * scale); + canvas.getContext('2d').drawImage(video, 0, 0, canvas.width, canvas.height); + + return canvas.toDataURL('image/jpeg', 0.7).split(',')[1]; +} + async function turn(said) { if (busy || ended) return; busy = true; heard = []; + $('heard').textContent = ''; lastTurn = Date.now(); if (said) show('me', said); status('Thinking…'); @@ -90,7 +147,7 @@ async function turn(said) { try { const { say, done } = await api(`/sessions/${session}/turn`, { method: 'POST', - body: JSON.stringify({ said }), + body: JSON.stringify({ said, screenshot: screenshot() }), }); if (say) { @@ -121,7 +178,7 @@ function listen() { function setupRecognition() { recognition = new Recognition(); - recognition.lang = $('lang').value; + recognition.lang = LANG; recognition.continuous = true; recognition.interimResults = true; @@ -156,7 +213,7 @@ function tick() { const text = heard.join(' ').trim(); if (text) { - const wait = /\?\s*$/.test(text) + const wait = QUESTION.test(heard.at(-1) ?? '') ? PAUSE_AFTER_QUESTION : PAUSE_AFTER_SPEECH; if (now - lastSpeech > wait) turn(text); @@ -201,6 +258,10 @@ async function startRecording() { }; recorder.start(10000); + screenVideo = document.createElement('video'); + screenVideo.muted = true; + screenVideo.srcObject = new MediaStream(screen.getVideoTracks()); + await screenVideo.play(); $('rec').textContent = '● recording'; // Stopping the share from the browser's bar ends the session. screen.getVideoTracks()[0].addEventListener('ended', () => finish()); @@ -238,7 +299,7 @@ async function start() { try { ({ id: session } = await api('/sessions', { method: 'POST', - body: JSON.stringify({ name: $('name').value, lang: $('lang').value }), + body: JSON.stringify({ name: $('name').value, lang: LANG }), })); await startRecording(); } catch (error) { @@ -274,3 +335,7 @@ $('agree').addEventListener('change', () => { }); $('start').addEventListener('click', start); $('end').addEventListener('click', () => finish()); +// Hands over at once, with whatever was heard so far. +$('ask').addEventListener('click', () => { + if (!speaking) turn(heard.join(' ').trim()); +}); diff --git a/usertest/page/index.html b/usertest/page/index.html index 54238486..8b0b3975 100644 --- a/usertest/page/index.html +++ b/usertest/page/index.html @@ -110,8 +110,9 @@

Try Atomic's new plugins

Google's speech service.
  • - That text and the app's error log are sent to Anthropic's Claude - API to decide what the moderator says next. + That text, a screenshot of your shared screen at each turn, and + the app's error log are sent to Anthropic's Claude API to decide + what the moderator says next.
  • If you connect your own accounts (for example Google Calendar), @@ -125,11 +126,7 @@

    Try Atomic's new plugins

    - - +

    The session is in English.

    Try Atomic's new plugins

    +

    + + or just pause after a question +

    From afe980ad3e4077ff0965053c9d8679cfbe8d9342 Mon Sep 17 00:00:00 2001 From: Michiel de Jong Date: Mon, 28 Sep 2026 15:24:08 +0200 Subject: [PATCH 4/4] usertest: ask testers to share their entire screen; stay silent after 'wait' Co-Authored-By: Claude Opus 5.5 --- usertest/moderator/script.md | 1 + usertest/page/index.html | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/usertest/moderator/script.md b/usertest/moderator/script.md index 88c126f7..aa26c139 100644 --- a/usertest/moderator/script.md +++ b/usertest/moderator/script.md @@ -10,6 +10,7 @@ Find out where people get stuck, what they expect, and what confuses them. The p - Speak English only, even if the tester answers in another language: the voice and the speech recognizer are set to English. If they speak another language, kindly ask them to continue in English. - Ask open questions: "What do you expect to happen?", "What are you looking for now?", "What do you make of that?" Never ask leading questions. - Encourage thinking aloud. If they fall silent (you get "(silence)"), gently ask what they are doing or thinking. +- If they ask you to wait, say "Sure" once, then answer [WAIT] to every "(silence)" until they speak again. - Mostly listen. When the tester is just thinking aloud and making progress, answer with only the token [WAIT]: nothing is spoken, and you get their next words later. Speak when they ask you something, finish or give up a task, fall silent, or the [Log] shows an error. - Don't explain how the app works and don't say which button to press. Only when the tester has been stuck on the same step for several turns and asks for help, give the smallest possible hint, and note that you did. - If the [Log] shows an error, don't name it. Ask what they see on the screen and what they expected instead. diff --git a/usertest/page/index.html b/usertest/page/index.html index 8b0b3975..a2205601 100644 --- a/usertest/page/index.html +++ b/usertest/page/index.html @@ -91,6 +91,11 @@

    Try Atomic's new plugins

    When you start, a second window opens with your own empty Atomic drive. Put it next to this one; keep this window open.
    • +
  • + When Chrome asks what to share, choose your + entire screen, so the moderator sees the Atomic + window too. +