usertest: the user-testing droplet's setup and log collector

.gitignore

@@ -7,3 +7,4 @@ node_modules
 /integrations/tooling/test-results
 playwright-report/
 /integrations/*/app/dist
+usertest/out/

usertest/Caddyfile

@@ -0,0 +1,30 @@
+# Caddy config for the user-testing droplet. Installed as
+# /etc/caddy/Caddyfile by deploy.sh; Caddy fetches the Let's Encrypt
+# certificates itself. BASE_DOMAIN and ACME_EMAIL come from
+# /etc/caddy/usertest.env, which the systemd drop-in caddy-usertest.conf
+# loads (see README.md).
+{
+	email {$ACME_EMAIL}
+}
+
+# atomic-server (server.sh), plain HTTP on loopback. It builds its links from
+# ATOMIC_DOMAIN and reads the scheme from X-Forwarded-Proto, which Caddy sets.
+plugins.{$BASE_DOMAIN} {
+	reverse_proxy 127.0.0.1:8080
+}
+
+# The test catalog and the drive app modules it points at (catalog.mjs).
+# The data-browser fetches the catalog cross-origin; app modules are
+# downloaded by the host and checked against their SRI hash.
+catalog.{$BASE_DOMAIN} {
+	root * /srv/catalog
+	header Access-Control-Allow-Origin *
+	header Cache-Control "no-cache"
+	file_server
+}
+
+# The log collector (collector/server.mjs): Sentry envelopes from
+# atomic-server and the data-browser, and /log posts from drive apps.
+logs.{$BASE_DOMAIN} {
+	reverse_proxy 127.0.0.1:8081
+}

usertest/README.md

@@ -0,0 +1,124 @@
+# User-testing instance
+
+An atomic-server test instance, where people try the drive apps on their own
+laptop and with their own provider accounts. It replaces sessions on one
+prepared laptop. It runs on one DigitalOcean droplet, set up on 2026-09-28
+(Ubuntu 24.04, 2 vCPU, 4 GB, AMS3). This folder holds everything needed to
+rebuild it.
+
+| Host | What |
+| --- | --- |
+| `https://plugins.<base-domain>` | atomic-server, the published e2e image of the pinned commit (`server.sh`) |
+| `https://catalog.<base-domain>/catalog.json` | the test catalog and the app modules it points at (`catalog.mjs`) |
+| `https://logs.<base-domain>` | the log collector (`collector/`) |
+| `https://localthought.io` | the integration proxy, shared with everyone else (not on the droplet) |
+
+The base domain is currently `178-62-223-35.sslip.io`. sslip.io resolves
+`<anything>.<a-b-c-d>.sslip.io` to the IP address `a.b.c.d`, so no DNS setup
+is needed. Drives are tied to the host name. Moving to a real domain later
+means fresh drives.
+
+## What differs from the published catalog
+
+`catalog.mjs` starts from `integrations/catalog.json` and makes every drive
+app installable: Google Calendar, GitHub issues, Bank statements, Clockify,
+Notion and Pets. It also sets `experimental: false`, so testers need no
+toggle. The published catalog keeps these disabled until launch. Apps not yet
+in `apps/` are built from this checkout and served from the droplet. Pets
+uses its published module.
+
+Known limits, as of 2026-09-28:
+
+- Pets cannot connect: localthought.io has no `pets` platform (#174).
+- The calendar app does not import recurring events.
+- The e2e image exposes test-only routes (`/app/prunetests`,
+  `/app/sandbox`).
+- `ATOMIC_HOST_MODE=open`: anyone with the URL can create a drive. Stop the
+  container between session days.
+
+## Setting it up from scratch
+
+On a fresh Ubuntu 24.04 droplet, as root:
+
+```sh
+apt-get update && apt-get install -y docker.io caddy
+ufw allow 22/tcp && ufw allow 80/tcp && ufw allow 443/tcp && ufw --force enable
+printf 'BASE_DOMAIN=%s\nACME_EMAIL=%s\n' 178-62-223-35.sslip.io you@example.org > /etc/caddy/usertest.env
+mkdir -p /var/lib/usertest-logs
+```
+
+Docker publishes the server on `127.0.0.1` only, so ufw's rules are not
+bypassed.
+
+From a checkout of this repo, with the layout from `AGENTS.md`
+(`node integrations/tooling/link-atomic-server.mjs`) and each app's
+dependencies installed:
+
+```sh
+for a in calendar money notion timesheets; do (cd integrations/$a && pnpm install); done
+(cd integrations/issue-tracker/app && pnpm install --frozen-lockfile)
+USERTEST_LOG_URL=https://logs.178-62-223-35.sslip.io/log node usertest/catalog.mjs
+sh usertest/deploy.sh root@178.62.223.35
+ssh root@178.62.223.35 sh /opt/usertest/collector/run.sh
+ssh root@178.62.223.35 sh /opt/usertest/server.sh 178-62-223-35.sslip.io
+```
+
+## Updating an app
+
+1. Change the app, then bump its entry in `VERSIONS` in `catalog.mjs`.
+2. Run `USERTEST_LOG_URL=https://logs.178-62-223-35.sslip.io/log node usertest/catalog.mjs`,
+   then `sh usertest/deploy.sh root@178.62.223.35`.
+3. On Integrations, testers who installed the old version see "Update to
+   <version>".
+
+Never rebuild into an existing version: the host refuses a module whose bytes
+no longer match the hash in the catalog.
+
+## For testers
+
+1. Open `https://plugins.<base-domain>/app/dev-drive`. It creates an agent
+   and a drive in the browser, without signup.
+2. In Settings → Integration, set the plugin catalog URL to
+   `https://catalog.<base-domain>/catalog.json`.
+3. On Integrations, install an app and connect it with your own account.
+
+Step 2 goes away once the planned `/usertest` page sets it.
+
+## Logs
+
+The collector (`collector/server.mjs`, no dependencies, run in a
+`node:22-alpine` container by `collector/run.sh`) writes one JSON line per
+event to `/var/lib/usertest-logs/<UTC date>.jsonl`. It has two entrances:
+
+- **Sentry's envelope endpoint.** `server.sh` sets `SENTRY_DSN` (project 1,
+  atomic-server) and `SENTRY_DSN_BROWSER` (project 2, the data-browser, which
+  atomic-server injects into the page at runtime). This reports uncaught
+  errors in the page and the server's `error!` events. It also switches on
+  the sidebar's Feedback form, whose messages arrive as `type: feedback`.
+- **`POST /log`**, a JSON object or an array of them, for anything else. A
+  drive app's errors stay inside its sandboxed frame: the host shows them
+  there and reports nothing. The calendar app hands its failures, a line
+  per finished sync and anything uncaught to a hook (`app/report.ts`), and
+  `catalog.mjs` prepends the hook that posts here when `USERTEST_LOG_URL`
+  is set. The app itself makes no network request.
+
+Verified on 2026-09-28: an error thrown in the data-browser arrived within
+seconds with its stack and URL.
+
+```sh
+ssh root@178.62.223.35 'tail -f /var/lib/usertest-logs/$(date -u +%F).jsonl'
+ssh root@178.62.223.35 docker logs --since 1h atomic-plugins
+heroku logs -a integration-proxy -n 500   # needs access to the Heroku app
+```
+
+Both endpoints are public, like any Sentry DSN, and nothing checks who
+sends. Bodies over 1 MB are refused.
+
+## Privacy
+
+Testers connect real accounts. Their data is on the droplet, in the Docker
+volume `atomic-plugins-store`, and their provider tokens are in
+localthought.io's database. Error reports can contain what was on screen
+(titles in messages, URLs), and they stay in `/var/lib/usertest-logs`. Tell testers this before they start. Reset the
+store with `docker rm -f atomic-plugins && docker volume rm atomic-plugins-store`,
+then run `server.sh` again.

usertest/caddy-usertest.conf

@@ -0,0 +1,5 @@
+# systemd drop-in, installed by deploy.sh as
+# /etc/systemd/system/caddy.service.d/usertest.conf: gives Caddy the
+# variables the Caddyfile reads.
+[Service]
+EnvironmentFile=/etc/caddy/usertest.env

usertest/catalog.mjs

@@ -0,0 +1,135 @@
+#!/usr/bin/env node
+/**
+ * Builds the user-testing catalog into a folder that deploy.sh copies to the
+ * droplet's /srv/catalog:
+ *
+ *   node usertest/catalog.mjs [out]      # default out: usertest/out
+ *
+ * With USERTEST_LOG_URL set (deploy it with
+ * USERTEST_LOG_URL=https://logs.<base-domain>/log), the modules of apps
+ * marked `report: true` start with a prelude that defines `globalThis.__USERTEST_REPORT__`, which
+ * posts what an app hands it to the collector. Apps call it through their
+ * own `report.ts` (calendar so far). The apps themselves make no network
+ * request of their own (their build tests check for `fetch(`); only this
+ * prelude does, and only in these test builds.
+ *
+ * It starts from this checkout's integrations/catalog.json and makes every
+ * drive app installable and visible without the "Show experimental plugins"
+ * toggle, including the ones the published catalog keeps disabled until
+ * launch. Apps not yet published to apps/ are built from this checkout with
+ * their own app/build.mjs and served next to the catalog, at
+ * apps/<id>/<version>/ui.js, with the SRI hash the host checks on install.
+ * Pets keeps its published module.
+ *
+ * VERSIONS below is the only thing to edit. Bump an app's version whenever
+ * its build changes: the host offers "Update to <version>" only for a new
+ * version string, and a changed file under an old version fails the
+ * integrity check for anyone who installs it afterwards.
+ *
+ * Needs the layout AGENTS.md describes (browser/ from the pinned
+ * atomic-server) and each app's dependencies installed (see README.md).
+ */
+import { createHash } from 'node:crypto';
+import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
+import { dirname, resolve } from 'node:path';
+import { fileURLToPath, pathToFileURL } from 'node:url';
+
+const here = dirname(fileURLToPath(import.meta.url));
+const repo = resolve(here, '..');
+const out = resolve(process.argv[2] ?? resolve(here, 'out'));
+
+const A = 'https://atomicdata.dev/properties/';
+const I = 'https://atomicdata.dev/integrations/properties/';
+
+/** Drive apps built here. `base` is the catalog entry whose copy they reuse. */
+const VERSIONS = {
+  calendar: 'usertest-3',
+  'issue-tracker': 'usertest',
+  money: 'usertest',
+  notion: 'usertest',
+  timesheets: 'usertest',
+};
+const APPS = {
+  calendar: {
+    base: 'devonian-google-calendar',
+    name: 'Google Calendar',
+    emoji: '📅',
+    row: ['Event', 'Events'],
+    // Has app/report.ts; gets the collector prelude.
+    report: true,
+  },
+  'issue-tracker': {
+    base: 'devonian-todoist',
+    name: 'GitHub issues',
+    emoji: '🐙',
+    description: 'Import the issues of a GitHub repository into a table.',
+    row: ['Issue', 'Issues'],
+  },
+  money: { base: 'money', row: ['Transaction', 'Transactions'] },
+  notion: { base: 'notion', row: ['Page', 'Pages'] },
+  timesheets: { base: 'timesheets', row: ['Time entry', 'Time entries'] },
+};
+
+const LOG_URL = process.env.USERTEST_LOG_URL;
+if (LOG_URL && !/^https:\/\/[^/]+\/log$/.test(LOG_URL))
+  throw new Error('USERTEST_LOG_URL must look like https://<host>/log');
+/** text/plain keeps the post a simple request: no CORS preflight from the
+ * frame's null origin. A failing collector never affects the app. */
+const PRELUDE = LOG_URL
+  ? `globalThis.__USERTEST_REPORT__=e=>{try{fetch(${JSON.stringify(LOG_URL)},{method:"POST",keepalive:!0,headers:{"content-type":"text/plain"},body:JSON.stringify(e)}).catch(()=>{})}catch{}};\n`
+  : '';
+
+const catalog = JSON.parse(
+  readFileSync(resolve(repo, 'integrations/catalog.json'), 'utf8'),
+);
+const byShortname = name => catalog.find(r => r[A + 'shortname'] === name);
+
+for (const [id, app] of Object.entries(APPS)) {
+  const version = VERSIONS[id];
+  const { build } = await import(
+    pathToFileURL(resolve(repo, 'integrations', id, 'app/build.mjs')).href
+  );
+  const file = resolve(out, 'apps', id, version, 'ui.js');
+  mkdirSync(dirname(file), { recursive: true });
+  await build({ outfile: file });
+  if (PRELUDE && app.report)
+    writeFileSync(file, PRELUDE + readFileSync(file, 'utf8'));
+  const bytes = readFileSync(file);
+
+  const source = byShortname(app.base);
+  if (!source) throw new Error(`catalog.json has no entry ${app.base}`);
+  const entry = { ...source };
+  entry[A + 'localId'] = id;
+  entry[A + 'shortname'] = id;
+  if (app.name) entry[A + 'name'] = app.name;
+  if (app.emoji) entry[A + 'emoji'] = app.emoji;
+  if (app.description) entry[A + 'description'] = app.description;
+  // A drive app, not a sandbox plugin: it needs no API-plugins host.
+  delete entry[I + 'requires-api-plugins'];
+  entry[I + 'version'] = version;
+  entry[I + 'app-module'] = `apps/${id}/${version}/ui.js`;
+  entry[I + 'app-module-integrity'] =
+    'sha384-' + createHash('sha384').update(bytes).digest('base64');
+  entry[I + 'app-row-name'] = app.row[0];
+  entry[I + 'app-row-name-plural'] = app.row[1];
+
+  const at = catalog.indexOf(source);
+  if (app.base === id) catalog[at] = entry;
+  else catalog.splice(at + 1, 0, entry);
+}
+
+// Every drive app, Pets included: enabled, and shown without the toggle.
+for (const entry of catalog) {
+  if (!entry[I + 'app-module']) continue;
+  entry[I + 'enabled'] = true;
+  entry[I + 'experimental'] = false;
+}
+
+mkdirSync(out, { recursive: true });
+writeFileSync(
+  resolve(out, 'catalog.json'),
+  JSON.stringify(catalog, null, 2) + '\n',
+);
+for (const entry of catalog)
+  if (entry[I + 'app-module'])
+    console.log(`${entry[A + 'shortname']} ${entry[I + 'version']}`);

usertest/collector/run.sh

@@ -0,0 +1,12 @@
+#!/bin/sh
+# (Re)starts the log collector (server.mjs) on the droplet, as root:
+# `sh run.sh`. Logs go to /var/lib/usertest-logs/<UTC date>.jsonl; follow a
+# session with `tail -f /var/lib/usertest-logs/$(date -u +%F).jsonl`.
+set -eu
+HERE=$(cd "$(dirname "$0")" && pwd)
+docker rm -f usertest-collector >/dev/null 2>&1 || true
+docker run -d --name usertest-collector --restart unless-stopped --init \
+  -p 127.0.0.1:8081:8081 \
+  -v "$HERE/server.mjs:/app/server.mjs:ro" \
+  -v /var/lib/usertest-logs:/logs \
+  node:22-alpine node /app/server.mjs