diff --git a/AGENTS.md b/AGENTS.md index 9b331c520e..17a6b89e49 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -44,6 +44,16 @@ second writer on tracked files. Four ways that has cost real time: vite sees a file change, and the page reloads. That is not cosmetic: a reload abandons in-flight work, and this silently cancelled a destroy mid-test. If `git status` shows `.po` churn on a clean checkout, settle them and commit. +- **The compiled catalogs are a second artifact.** Wuchale compiles the `.po` + files into `src/locales/.wuchale/`, which is gitignored. `git checkout -- + src/locales/` resets the `.po` files and leaves the compiled output from + another state; the two then disagree and *every indexed string shifts*. + Missing entries show as `[i18n-404:NNN]`; shifted entries show a different + real string, so nothing looks wrong and tests fail on the wrong element. + Whenever you reset the `.po` files, also `rm -rf + browser/data-browser/src/locales/.wuchale` and restart vite. Do not make + the `.po` files read-only to stop the rewrite; the compiled output goes + stale instead. - **`pnpm clean-translations` is not the same writer.** It extracts from test files too, which the vite plugin does not, so its output is a *different* fixed point. Settling by running the app is what matches the dev server. @@ -283,6 +293,13 @@ cd browser && pnpm run test-e2e:light # Playwright @smoke (feature-br cd browser && pnpm run test-e2e # Full Playwright suite (develop / tags) ``` +When you restart an `atomic-server` for tests, gate on `curl` returning HTTP +200, never on the port: redb's lock outlives the listening socket by several +seconds, so an immediate restart exits with "Database already open" and the +suite runs against nothing, failing in ways that look like product bugs. Retry +the spawn when the log says so, and give a fresh server about ten seconds +before the first spec. + `atomic_lib`'s unit tests need the `db` feature — `hierarchy.rs`'s test module calls `Db::init_temp` and `test_utils::setup_test_env`, both gated behind it. So `cargo test -p atomic_lib --no-default-features` does **not** compile; don't diff --git a/planning/E2E_HANDOFF_1500.md b/planning/E2E_HANDOFF_1500.md deleted file mode 100644 index 37c7fb6ae7..0000000000 --- a/planning/E2E_HANDOFF_1500.md +++ /dev/null @@ -1,365 +0,0 @@ -# Handoff: e2e suite for PR #1500 - -Updated 2026-09-16 (second session). Goal unchanged: get the Playwright suite -green for `codex/website-self-hosted-publishing`, then **Rebase and merge**. - -## The most important correction to the previous handoff - -Most of the "unexplained" website failures (group 4) and a good share of the -rest were **not product bugs — there was no server running**. - -`redb`'s lock outlives the listening socket. Killing the server frees port 9897 -within a second, but `atomic.redb` stays locked for several more, so a restart -that only waits for the port exits immediately with -`Error: Failed to create redb ...: Database already open. Cannot acquire lock.` -The suite then runs against nothing and every spec fails in ways that look like -real defects: missing menu items, disabled buttons, empty panels. - -**Never gate on the port. Gate on `curl` returning HTTP 200, and retry the -spawn — not just the wait — when the log says "Database already open".** -`scripts/start-server.sh` (below) does this. Also give a freshly started server -~10s before running specs; within that window `menu-item-website-prepare` -renders but stays `disabled`. - -## Stack setup (delta from the previous handoff) - -Everything in the previous handoff still applies, plus: - -1. **Patched loro-prosemirror.** `data-browser/node_modules` resolves - loro-prosemirror out of `~/dev/atomic-server-plugin-model`, where - `browser/patches/loro-prosemirror@0.4.3.patch` is **not** applied (that - checkout's `.modules.yaml` has no `patchedDependencies`). CI applies it. - Without it the inline editors drop characters and - `website-inline-content` / `website-inline-rte` fail. Fix, touching neither - of Joep's checkouts: - - copy the resolved package to `/private/tmp/lp-patched/node_modules/loro-prosemirror` - - `/opt/homebrew/bin/git apply` the patch there - - symlink its five sibling deps (`lib0`, `loro-crdt`, `prosemirror-model`, - `-state`, `-view`) beside it so its own imports resolve - - alias `loro-prosemirror` to that `dist/index.js` in - `/private/tmp/hosting-vite.config.mts` and add `/private/tmp/lp-patched` - to `server.fs.allow` - - `rm -rf /private/tmp/hosting-vite-cache` and restart Vite, or the stale - pre-bundle is still served - -2. **Mock integration proxy**, matching `atomicService` in `.dagger/src/index.ts`: - ``` - cd integrations/localthought && MOCK_FRONTEND_ORIGIN=http://localhost:6763 \ - MOCK_PROXY_HOST=127.0.0.1 MOCK_PROXY_PORT=19090 \ - TENANT_SECRET='bW9jay10ZW5hbnQ.mock-signature' node mock-proxy.mjs - ``` - The server wants `ATOMIC_INTEGRATION_PROXY_URL`, `TENANT_SECRET` and - `ATOMIC_INTEGRATION_FRONTEND_ORIGIN`; Vite wants `VITE_E2E=true` and - `VITE_INTEGRATION_PROXY_URL=http://127.0.0.1:19090`. - `ATOMIC_MOCK_INTEGRATION_PROXY=1` on the Playwright side only gates one - `test.skip` in `plugins.spec.ts` — it cannot affect any other spec. - -3. **create-template** needs building and its own `node_modules`: - `../node_modules/.bin/tsc` in `browser/create-template` emits - `bin/src/index.js` (type errors are expected without deps), and - `browser/create-template/node_modules` can symlink to the one in - `/private/tmp/atomic-assistant-website`, matching the other packages. - -4. **Worker count matters.** At `PLAYWRIGHT_WORKERS=4`, `documents.spec.ts`, - `username-live.spec.ts` and two `apps.spec.ts` tests fail and then pass at 2 - workers. Treat 4-worker-only failures as contention until reproduced lower. - -## Fixed this session (commit `6e79f6a4e`) - -- **Test bug class, the right panel is transient (#1475).** - `RightPanelProvider` keeps open/closed in `useState` and deletes - `atomic.rightPanel.active`; `atomic.sidebar-panels` only controls which - panels are *available*. Specs must click the navbar toggle — - `openAISidebar(page)` / `sendChatMessage(page, ...)` in `tests/ai-mock.ts`. - `website.spec.ts`, `assistant-file-drop.spec.ts` and the shared - `tests/legacy-github-setup.ts` (which broke both `app-setup.spec.ts` tests) - hand-rolled `page.locator('[data-open]')` instead. A bare `[data-open]` - locator is the smell; correct uses scope it to a testid. -- **`integration-workspace.spec.ts`**: `ConnectGitHub.tsx` became a thin - wrapper around the schema-driven `AppSetupForm` and no longer names the - installer path the spec regexes out of the served module — read - `githubInstaller.ts` instead. The button is labelled "Edit with AI". -- **`browser/lib/src/websockets.ts`**: guard `importEnvelopes` the way - `client-db.node.ts` does. `getClientDb()` can return an implementation - without it, and the unguarded call threw a TypeError out of the SYNC_PUSH - handler, taking the rest of that frame's handling with it. This was the one - real product defect the previous session identified; `onboarding-storage` - passes now. **Rebuild `browser/lib/dist` (`pnpm exec tsup`) after changing - lib, and restart Vite.** - -## Where the numbers stand - -Two full runs on a correct stack, both **236 passed / 42 failed / 7 skipped** -(the previous session's baseline was 231/46/8). The headline is the same at 4 -and 2 workers but the *composition* differs — `apps.spec.ts` contributed 4 -failures in one run and 6 in the other, `documents.spec.ts` and -`username-live.spec.ts` failed only at 4 workers. A meaningful share of the 42 -is flake, so treat any single run's list as provisional and reproduce before -chasing. - -Latest run, by spec: plugins 11, apps 6, table-create-perf 3, -vault-backup-restore 2, template 2, recovery-option 2, meetings 2, -google-calendar-import 2, file-upload-offline 2, drive-template-onboarding 2, -and one each in website-inline-rte, second-device-load, opfs-init-perf, -offline-tables, kanban, integration-workspace, integration-visibility, -devonian-issue-sync. - -`@smoke` — the only gate CI runs — is 17/18, the exception being the -order-dependent `second-device-load` noted below. - -## Still failing — triage - -- **`plugins.spec.ts` (9), `google-calendar-import` (2), - `integration-visibility` (2), `devonian-issue-sync` (1).** Not yet diagnosed - individually. These now run against a real mock proxy, so what remains is - worth reading as real. -- **Diagnostics group: `offline-tables`, `file-upload-offline` (2), - `opfs-init-perf`, `recovery-option` (2), `vault-backup-restore` (2).** These - fail the fixture's zero-console-error gate, not their own assertions. The - error is always the same: the app fetches `http://localhost:9897/` and the - server 404s it. On this branch drives are `did:ad:drive:...` subjects and - **nothing lives at the server's HTTP root** — a fresh data dir with - `ATOMIC_INITIALIZE=true` 404s there too, so this is architectural, not - misconfiguration. Either the client should not fetch the root, or these - tests should declare the diagnostic. `fixtures.ts` has no global allowlist by - design ("expected failures belong to the test that causes them"). -- **`integration-workspace.spec.ts:159`** — genuine product suspicion. After - "New automation", `WorkspaceControls` calls `onStart={() => close()}` and - `askAI(...)`, but the `` stays open with its - content unmounted, and it is the only thing in the a11y tree. If - `handleClosed` never runs, `useDialog` also leaves `inert` on ``, - which would make the app unclickable. Worth chasing in - `components/Dialog/index.tsx` (`finishClose` / the `data-closing` effect). -- **`apps.spec.ts`** is the worst flake in the suite: 4 failures in one full - run and 6 in the next, a different test at 1 vs 2 workers, always waiting on - `iframe[title="App"]`. Look at what gates that iframe before reading any - individual failure as real. -- **`second-device-load.spec.ts` (@smoke)** fails standalone but passed inside - the OPFS group run — order-dependent. Green on CI. -- **`template.spec.ts` (2)** should be unblocked by the build above, but each - test scaffolds and builds a real Next/SvelteKit site and is very slow. -- **Not investigated:** `table-create-perf` (3, perf thresholds), `kanban:177`, - `meetings` (2), `drive-template-onboarding` (2). - -## Merge posture - -CI's e2e gate is `@smoke` only (~18 tests) and is green. The full suite has -never been green on this branch, and several of the failures above are stale -tests or missing declarations in code the PR itself introduces — they are not -regressions against `develop`. Deciding how many of them must be green before -merging is Joep's call. - ---- - -# Second pass: where the failures actually come from - -Prompted by "develop is 0 failed" — that is **not reproducible in this -environment**, and the comparison needs care. - -## The local harness was never the CI harness - -**The checkout's dependencies came from a different branch.** Every -`node_modules` under `browser/` was a symlink into -`/private/tmp/atomic-assistant-website` (which itself resolves into -`~/dev/atomic-server-plugin-model`). This branch's own `pnpm-lock.yaml` had -never been installed. Consequences, all of which looked like product bugs: - -- `browser/patches/loro-prosemirror@0.4.3.patch` was not applied (the manual - copy + Vite alias described above is no longer needed — a real install - applies it, and the result is byte-identical). -- A **hard WASM crash** in the production build: `[ClientDb Worker Error] - Uncaught Error: null pointer passed to rust`, which broke every `@smoke` - test (18/18 failing). - -Fix: remove the symlinks (never `rm -rf` — they point at other checkouts) and -`pnpm install --frozen-lockfile` in `browser/`. It takes ~8s. Several workspace -packages must then be built before the data-browser will bundle: -`@tomic/service-ui`, `@tomic/plugin`, `@tomic/react`, `@tomic/lib` -(`pnpm --filter build`). After that the prod `@smoke` run went 18 failed -→ **2 failed, 16 passed**. - -**CI serves a production build, not Vite.** `.dagger/src/index.ts` builds -`data-browser/dist`, copies it to `server/assets_tmp`, and the server embeds -and serves it; Playwright's `FRONTEND_URL` is the server itself. Reproduce with: - -``` -cd browser/data-browser && SKIP_WASM_BUILD=1 VITE_E2E=true \ - VITE_INTEGRATION_PROXY_URL=http://127.0.0.1:19090 pnpm build -cd ../.. && rm -rf server/assets_tmp && mkdir -p server/assets_tmp \ - && cp -R browser/data-browser/dist/. server/assets_tmp/ -cargo build -p atomic-server # with the CLT env; build.rs embeds assets_tmp -# then FRONTEND_URL=http://localhost:9897 -``` - -## Measured, on a correct install - -| Stack | Result | -|---|---| -| Vite dev, correct deps | 230 passed / **47 failed** / 7 skipped | -| Production build (CI topology) | 229 passed / **49 failed** / 7 skipped | -| `@smoke` on the production build | 16 passed / 2 failed | - -The *number* barely moved; the *composition* did (prod fixes meetings, -table-create-perf, opfs-init-perf, template, recovery-option and breaks the -website specs — the website preview iframe wants an origin separate from the -app, which the same-origin prod topology does not give it). Failures stable -across all three topologies are the real ones. - -## Attribution — the answer to "develop is 0 failed" - -Of the failing specs, **these do not exist on develop at all**: `plugins` -(11), `apps` (6), `google-calendar-import` (2), `integration-workspace` (2), -`integration-visibility` (2), `drive-template-onboarding` (2), -`devonian-issue-sync` (1), `website-inline-rte` (1). That is **~27 of 47** — -new tests for new features, which develop cannot fail because it does not have -them. `develop` being green says nothing about them. - -Four failing specs are **byte-identical** to develop: `meetings`, -`second-device-load`, `opfs-init-perf`, `offline-tables`. Run against a -develop stack built in this same environment (worktree at -`/private/tmp/atomic-develop-base`, server on 9898, Vite on 6764), **all of -them fail there too** — 7 failures on develop versus 4 on the branch. So they -are not branch regressions. (That develop stack has its own setup gap — -`/app/dev-drive` times out for most specs — so treat it as evidence about -these four specs, not as a full baseline.) - -Conclusion: the branch is not ~47 regressions deep. The bulk is new-feature -tests that have never passed, plus environment and flake. What genuinely needs -work before "green" is the new-feature specs, led by `plugins` and `apps`. - -## `apps.spec.ts` — two real bugs found (6 failures) - -1. **A freshly created app renders its generic resource page, not its own - view; a reload fixes it.** `ResourcePage` gates the app view on - `appClass !== undefined && resource.hasClasses(appClass)`, and `appClass` - comes from `useDriveClass` in `chunks/PluginRuns/runScript.ts`. That hook - reads the drive's `defaultOntology` once and subscribes to it — but a drive - with no plugin schema yet **has no `defaultOntology`**, so it subscribes to - nothing. Creating the first app mints the schema and sets the property, and - the hook never hears about it; `appClass` stays `undefined` until a - remount. Fix direction: also subscribe to the drive resource and - re-subscribe when the ontology appears. (Written and reverted here — it is - correct on its own terms but bug 2 masks any test-visible benefit, so it - should land with a test that proves it.) -2. **App creation itself races.** In some runs `createApp` completes and - navigation lands on the app (verified by instrumenting the action: - `run:start → createApp:done → handOver:ok → navigate:`). In others the - URL stays on the drive and `findSchema(store, drive, pluginSchema())` - returns a schema with an **empty `classes` map**. Same code, same stack — - so `ensureSchema` has a race. This is the one to chase first. - ---- - -# Third session, 2026-09-17 — 41 failures down to 14 - -Full suite on the local stack (Vite dev, 2 workers): **265 passed / 14 failed / -7 skipped**, from 237/41/7 at the start of the session. - -## The operational finding that mattered most - -**The server degrades as its data dir grows.** After a day of runs -`/private/tmp/atomic-hosting-node/data` had reached **1.1 GB**, and on that -server all six `apps` tests and most of `plugins` failed on a 10s -`iframe[title="App"]` wait — which reads exactly like a product bug and is not -one. Moving the data dir aside and restarting took `apps` + `plugins` at two -workers from **16 failures to 1**. Do this before any full run; -`scripts/start-server.sh` in the session scratchpad starts the server and gates -on HTTP 200 (never on the port — redb's lock outlives the socket). - -Everything else about the stack is as the second session described. Firefox was -missing from the Playwright cache (`pnpm exec playwright install firefox`); -`client-db-locks` passes with it. - -## Fixed — product - -- `027c09094` **The empty string is not a subject.** `normalizeSubject('')` - resolves against the server URL, so `useResource(x ?? '')` and every render - before the drive setting hydrated fetched `http://localhost:9897/`. Nothing - lives at the root on a DID-drive branch, so each of those 404s and logged two - console errors — failing the zero-diagnostics gate in `offline-tables`, - `sync`, `file-upload-offline` and `drive-deeplink @smoke`. Guarded in - `getResourceLoading` and the async `getResource`. `CustomViewProvider` had - the same shape one level up (`/plugin-list?drive=`). -- `d1e91ddba` **The app-setup dialog could not close.** It derived `show` from - `useDialog`'s post-animation `isOpen`, which only goes false once the Dialog - has seen `show` go false: a deadlock, with the modal swallowing every click - behind it. That was `plugins.spec.ts:893`. -- `e4141c008` **A regression from #1510.** Reusing a same-shortname select - property threw when it lacked a requested option, and the Student and - Personal templates each pair a task "Status" with a reading-list "Status". - The whole gallery died with `has no option "Want to read"`. Now it - disambiguates to `status-2`, as the incompatible-datatype branch already did. - -## Fixed — tests - -- `2bf4e927f` / `3e86c4be2` **Kanban drags.** The board FLIP-animates after a - drop, so the second drag pressed where the card used to be and dnd-kit never - activated. `await source.hover()` first; measure the drop target after - activation, not before. Same fix in the Devonian spec. -- `00414440d` `integration-visibility` used `check()` on a toggle that - disappears once checked; `integration-workspace` asserted "no dialog at all" - when the app legitimately raises the AI model-setup dialog. - -## What is still red, and why - -| Spec | Verdict | -|---|---| -| `recovery-option` (2), `opfs-init-perf`, `table-create-perf` (3) | Dev-topology only. The SW warning `recovery-option` declares happens only in a production build (VitePWA is off in dev); the `clientdb.*` perf marks are likewise absent from the dev module graph. All pass in CI's topology. | -| `vault-backup-restore` (2) | Points at a portal on :49237, which is **another checkout's** atomic-saas dev server. Needs this branch's portal to mean anything. | -| `google-calendar-import` (2) | Real gap. The new integration-visibility preference is only readable with the server reachable, and this spec runs the whole scenario with the server cut off. Enabling discovery before the interception starts lists the Calendar card; the reload that follows loses it again. Fix = make the preference and its schema readable from the local DB. | -| `second-device-load` @smoke | Real, pre-existing flake — ~1 in 3 passes locally, green on CI. Evidence: after unlocking, `document.body` has no `SecondDeviceChild`; **a reload makes it appear**, so the data is on the server and it is the post-unlock render that misses it. A `StoreEvents.AgentChanged` → `invalidateCollection` hook in `useCollection` was tried and did **not** fix it (reverted). Next place to look is what the sidebar's `useChildren` collection does between "locked, anonymous query" and "signed in". | -| `offline-chatroom`, `website-inline-rte` | Flaky; each passes when run alone. `website-inline-rte` is the known dropped-character flake in the preview editor. | - -Nothing in the remaining list is a regression this branch introduces, and the -`@smoke` gate CI runs is green apart from the `second-device-load` flake above. - -## Later the same night — 41 → 11, and three more real fixes - -Full suite, dev topology, 2 workers: **268 passed / 11 failed / 7 skipped**. - -- `f0de176c6` **A children page older than its own count.** - `getMemberWithIndex` returns undefined when the page it reads does not hold - the index the count implies; `useChildren` dropped that slot and never looked - again, so a child could be missing for the life of the view. - `second-device-load @smoke` went from ~1 pass in 3 to 6 in 7. The retry only - runs while the hook has never produced a list — after a delete the count runs - ahead of the page too, and refreshing there would pull the destroyed resource - back into the store. -- `993ad653b` **A child that did not exist when the query was answered.** - The collection has no reason to ask again, so a folder created moments - earlier stayed missing from the drive's list until a reload — - `server-only-fallback.spec.ts`, failing about half the time. `useChildren` - already listens for `ResourceUpdated` to re-sort; it now also re-reads the - query when the changed resource names this parent and is absent from the - list. That spec: 4/4. `e2e.spec.ts` "delete resource" @smoke, which had been - ~50/50 in this environment, also went 4/4. -- `8183a3bc6` **`google-calendar-import` (both tests) now pass.** They enabled - API-plugin discovery *after* cutting the server off, and that preference is a - write whose `/commit` the spec aborts. Set it up first, read the card back so - the preference and schema are in the local DB, flush, then go offline. The - offline phase must also open the drive by subject — `/app/dev-drive` - bootstraps the drive and, with no server, rebuilds the ontology empty. - -### What is left, honestly - -`vault-backup-restore` (2) needs its own control plane: `atomic-saas` -hardcodes port 3030, where Joep already runs one, and `vault-stack.sh` points -enrolled drives at the node in `.env.development` (9885 — the real dev server). -Not something to start from a session that must not disturb that stack. - -`table-create-perf` (3), `opfs-init-perf` and `recovery-option` (2) were -called "dev-topology only" above. That is right for `recovery-option` (the -service-worker warning it declares only happens in a production build), but a -production build made in this environment fails the other two *differently* -(`/app/dev-drive` never reaches a drive), so the prod topology here is not -clean enough to prove them either way. Treat them as unexplained rather than -explained. - -The rest is flake that passes when run alone: `offline-chatroom`, -`tables.spec.ts` Shift+Enter, `website-inline-rte` (the known dropped-character -flake in the preview editor). - -**CI is red for an unrelated reason.** Every Mancave run in the repo since -10:34 on 2026-09-16 fails in `Log in to Docker Hub` with `Error: spawn EIO`, -before any build step — `develop`, `feat/plugin-catalog` and three other -branches alike. The runner host needs attention; re-running does not help. diff --git a/planning/E2E_HANDOFF_FABLE.md b/planning/E2E_HANDOFF_FABLE.md deleted file mode 100644 index c5b40219b8..0000000000 --- a/planning/E2E_HANDOFF_FABLE.md +++ /dev/null @@ -1,248 +0,0 @@ -# e2e handoff — PR #1500 (`codex/website-self-hosted-publishing`) - -Written 2026-09-17. HEAD at handoff: `077555c32`. - -Goal: get the Playwright suite green so #1500 can merge (**Rebase and merge** — -merge commits are disabled and squash would collapse 167 commits including -Michiel's). - -Last measured full run: **269 passed / 5 failed / 4 flaky / 7 skipped** (19 min, -2 workers, retries=1). It was 41 failed at the start of the day. - ---- - -## 1. Read this first: the harness lies - -More of my wasted time came from the environment than from the code. Three traps, -in order of how much they cost: - -### 1.1 The i18n catalogs silently corrupt results - -The `wuchale` Vite plugin rewrites `browser/data-browser/src/locales/*.po` while -the dev server runs. The **compiled** catalogs live in `src/locales/.wuchale/`, -which is **gitignored**. So `git checkout -- src/locales/` restores the `.po` -files and leaves the compiled artifacts from some other state — the two disagree -and every indexed string lookup shifts. - -Two failure shapes; the second is the dangerous one: - -- missing entries render as `[i18n-404:2275]` — visibly broken -- shifted entries render as **a different real string** — nothing looks wrong - -That second one is how `iframe[title="Website preview"]` became -`"Preserves recurring series and previews supported edits to send back to -Google."` and every website spec failed. I bisected code for an hour before -finding it. - -**Before trusting any run**, regenerate both together and restart Vite: - -```bash -git checkout -- browser/data-browser/src/locales/ -rm -rf browser/data-browser/src/locales/.wuchale -# then restart Vite -``` - -Do **not** `chmod 444` the `.po` files to stop the rewrite — I tried; it freezes -them while `.wuchale` goes stale and `integration-visibility` then fails with -`[i18n-404]` toggles. The `.po` files going dirty *during* a run is fine as long -as both were generated in step. - -### 1.2 redb's lock outlives the listening socket - -After killing the server, port 9897 frees within a second but `atomic.redb` stays -locked for several more. Starting immediately gives `Database already open` and -the process exits — leaving **no server** while the suite runs, and every spec -fails in ways that look like product bugs. Gate on `curl` returning HTTP 200, not -on the port, and retry the spawn. `planning/start-server-9897.sh` does this. - -Also: give a freshly started server ~10s before running specs. - -### 1.3 Worker count and retries - -CI runs **2 workers per shard, 4 shards, each with its own atomic-server**, plus -retries (`.dagger/src/index.ts`, `HOST_PROFILES.mancave`). A local 4-worker run -against one server is much harsher and invents failures. Use -`PLAYWRIGHT_WORKERS=2 PLAYWRIGHT_RETRIES=1`. - -A rotating pair of specs fails per run purely from load (`plugins:909`, -`tables:482`, `drive-catalog`, `offline-tables`, … different each time). **Always -re-run a red in isolation before believing it.** - ---- - -## 2. Running the stack - -Checkout: `/private/tmp/atomic-pr-website`. Use `/opt/homebrew/bin/git` — the -Xcode git/python on PATH exit 69 (unaccepted licence). Rust builds need the -Command Line Tools env; see `project_rust_build_env.md` in the memory dir. - -**Dependencies are already installed correctly.** They were previously symlinked -from an unrelated branch's checkout, which is why the loro patch was missing and -why a production build died with `null pointer passed to rust`. If you ever need -to redo it: remove the symlinks (never `rm -rf` — they point at live checkouts), -`pnpm install --frozen-lockfile` in `browser/`, then build `@tomic/service-ui`, -`@tomic/plugin`, `@tomic/react`, `@tomic/lib` before bundling the data-browser. - -Three processes: - -```bash -# 1. server (9897) -bash planning/start-server-9897.sh - -# 2. mock integration proxy (19090) -cd integrations/localthought && MOCK_FRONTEND_ORIGIN=http://localhost:6763 \ - MOCK_PROXY_HOST=127.0.0.1 MOCK_PROXY_PORT=19090 \ - TENANT_SECRET='bW9jay10ZW5hbnQ.mock-signature' node mock-proxy.mjs & - -# 3. Vite (6763) — MUST have VITE_ATOMIC_SERVER_URL or it points at :9885, -# which is Joep's real dev server -cd browser/data-browser && VITE_ATOMIC_SERVER_URL=http://localhost:9897 \ - VITE_E2E=true VITE_INTEGRATION_PROXY_URL=http://127.0.0.1:19090 \ - node node_modules/.bin/../vite/bin/vite.js \ - --config /private/tmp/hosting-vite.config.mts --host localhost --port 6763 & -``` - -Verify after every Vite restart: - -```bash -curl -s http://localhost:6763/src/config.ts | grep -oE '"VITE_ATOMIC_SERVER_URL": "[^"]*"' -``` - -Run the suite: - -```bash -FRONTEND_URL=http://localhost:6763 SERVER_URL=http://localhost:9897 \ -WEBSITE_HOSTING_E2E=1 ATOMIC_MOCK_INTEGRATION_PROXY=1 \ -PLAYWRIGHT_WORKERS=2 PLAYWRIGHT_RETRIES=1 \ -browser/e2e/node_modules/.bin/playwright test \ - --config browser/e2e/playwright.config.ts --project chromium --trace off --reporter line -``` - -**Note on topology:** CI builds the data-browser and embeds it in the server -(`data-browser/dist` → `server/assets_tmp`), so Playwright hits the *server*, not -Vite. I've verified both work; the dev-server topology is what the numbers above -were measured on. Prod fixes some specs and breaks the website ones (the preview -iframe wants an origin separate from the app), so don't switch topology -mid-investigation and compare. - ---- - -## 3. What is actually failing - -### 3.1 `vault-backup-restore` ×2 — blocked on a missing service - -"Generate my recovery code" is clicked, the code never renders, and -`http://localhost:6763/api/recovery-secret` returns **404**. That is a -managed-node/portal API; a plain `atomic-server` does not serve it. The other -session's summary agrees: the vault panel needs the app to know a portal, and -only `atomic-managed-node` advertises one. - -Nothing to fix in the spec. Either run a managed node beside it or the spec needs -to declare the dependency. - -### 3.2 `website-inline-rte` — real, and the obvious fix is a dead end - -Symptom: ~50% failure, always a popup inside the preview iframe (mention list at -`:58`, link field at `:92`). The `@` trigger itself works 6/6 in isolation, so it -is state from earlier steps in the long sequence. - -**Diagnosis is confirmed by instrumentation.** During inline typing: - -| event kind | count | path | target | -|---|---|---|---| -| non-local (`import`) | 17/17 | `["properties"]` | `cid:root-properties:Map` | -| local | 18/18 | `["doc", …]` | — | - -The server echoes the author's own commit with a `lastCommit` stamp on the -resource's `properties` map. `LoroSyncPlugin` subscribes to the **doc**, not the -store (the store *does* suppress notify for an own-save echo — see -`commit-echo-and-save-cursor` in memory), so it rebuilds the whole ProseMirror -document for a change that never touches the document, and the rebuild destroys -any open popup. - -**Do not re-attempt the `updateNodeOnLoroEvent` filter.** The older notes call it -a "verified fix candidate". I implemented and measured it (skip non-local events -with no `path[0] === ROOT_DOC_KEY`, only when `containerId` is unset). Result: -`website-inline-rte` + `website-inline-content` went from a steady -1-failed-1-passed to **6 failed / 2 passed** over four runs, with two separate -failures — the mention list stops populating (the Suggestion popup really was -relying on those rebuild transactions) and the character drop at -`inline-content:107` persists anyway. I reverted it; the dependency is -byte-identical to its patched state and `browser/patches/` is untouched. - -The tractable framing is the **popup, not the patch**: make the tiptap Suggestion -plugin re-establish itself after a document rebuild, then the filter becomes -viable. That is product surgery in the editor — Joep has not green-lit it. - -Related and already committed: `a2503df27` clears the tiptap renderer reference -on destroy (`CommandsExtension` destroyed it on Escape and left the reference -set, so `onExit` destroyed it again). Genuine latent bug; does **not** fix this -flake, and the commit message says so. - -Iterating tip: edit the installed `dist/index.js` directly, then clear -`/private/tmp/hosting-vite-cache` and restart Vite. Vite pre-bundles the -dependency, so an edit is invisible until that cache is cleared — this cost me a -measurement that showed zero events. - -### 3.3 `plugins:909`, `tables:482` — not real - -Both pass in isolation (2/2 each). Load rotation, see §1.3. - ---- - -## 4. Fixed today (for context on what the causes look like) - -- `cc06e53ff` — `ensureAll` recovered each schema term with its own - `findByLocalId` **inside a loop**: 19 serialised `/query` round-trips, 13 of the - 14 seconds creating an app. Now issued together. `apps` 6→0, `plugins` 11→2. -- `077555c32` — `signIn()` decided "already signed in" from the settings link - being visible, which renders before the agent is in the store. Sessions with no - agent navigated on, read their drive unauthenticated, never adopted one, and so - never got a presence manager (it is keyed on the session drive). Agent state - correlated exactly with outcome. `presence-follow` + `meetings` 15/15. -- `82847e965` — spec edited `subjects[0]` of a `parent=` query assuming it was the - imported row; tables also hold draft placeholder rows with the same parent. -- `366156f53` — `AppSetupProvider` derived the Dialog's `show` from `isOpen`, - which only goes false *after* a close that needs `show` false. Deadlock: the - modal swallowed every click including its own close button. -- `6e79f6a4e` — specs hand-rolled `page.locator('[data-open]')` for the AI panel, - which is transient state and never restored (#1475). Use `openAISidebar` / - `sendChatMessage` from `tests/ai-mock.ts`. -- `a2f8f0125`, `45f721587` — action-menu snapshot race; Calendar spec read - `VITE_INTEGRATION_PROXY_URL` from the test process, but that is a build input - CI only passes to the bundler. - -## 5. Things I got wrong — don't redo them - -- `a5406e512` reverted `366156f53` on a measurement taken while the catalogs were - desynced. Reinstated. **Any measurement taken with `src/locales/` dirty is - suspect.** -- `9a1934038` retried the deep-link drive fetch on a 401, on a pending-handshake - theory. Reverted in `077555c32`: the session simply had no agent. -- The loro filter, above. - -All three shared a cause: acting on a 3–5 run sample in an environment I had not -yet established was trustworthy. Take more samples than feels necessary, and -re-check §1.1 before each. - -## 6. Coordination - -Another agent has been working **in this same checkout** on the same assignment -(`useChildren`, vault/portal, tombstones, perf-trace, optional diagnostics — -commits `8183a3bc6`, `993ad653b`, `0cd15fe9a`, `4117e9bd8`, `68af82756`, -`b9d03fcf7`, `fb56cfdba`, `f0de176c6`, …). Changing `node_modules` (a patch + -lockfile hash + reinstall) would swap dependencies under their running Vite, which -is why I left the loro patch alone. `366156f53` touches `AppSetupProvider`, close -to their integration work — likely conflict point. - -There is also a separate manual-testing stack for Joep on **6765** (server 9899, -data `/private/tmp/atomic-manual`) — leave it running. - -## 7. Suggested order - -1. Confirm §1.1 is clean, run the full suite once, and see which reds reproduce in - isolation. Expect ~2 of them not to. -2. `vault-backup-restore`: decide with Joep whether to run a managed node locally - or have the spec declare the dependency. Cheapest real win. -3. `website-inline-rte`: only worth starting if Joep green-lights the Suggestion - popup work. Do not touch the loro patch first. diff --git a/planning/README.md b/planning/README.md index efc0ad27c4..3b643e8745 100644 --- a/planning/README.md +++ b/planning/README.md @@ -31,26 +31,52 @@ now live in [`completed/`](./completed/): - [`trust-model-decision.md`](./completed/trust-model-decision.md) — the node that owns the URL is trusted with plaintext; anything that only stores is blind. - [`schema-routes-decision.md`](./completed/schema-routes-decision.md) — `did:ad:frozen` is the on-ramp, optional schema is the write-path policy. -## Biggest gaps (reconciled against code 2026-09-15) - -Ranked by impact. Each links to the plan that owns the work. - -1. ~~**Flutter writes are not durable.**~~ Fixed 2026-09-15: `Db::init_redb_file` owns the durable-flush tick, so every file-backed binding has the same 100ms loss bound. [`atomic-lib-runtime.md`](./atomic-lib-runtime.md). -2. ~~**No rate limiting on any write endpoint**~~ (fixed 2026-09-15, `server/src/rate_limit.rs`); the managed-node bootstrap grace still has no reaper. [`foss-public-host-mode.md`](./foss-public-host-mode.md) Phase 3, [`security-audit-2026-09.md`](./security-audit-2026-09.md) D, [`cloud-sync-managed-node.md`](./cloud-sync-managed-node.md). -3. ~~**Signed envelopes do not replicate**~~ Fixed 2026-09-15: envelopes ride in `SYNC_PUSH` and vault packs, verified on receipt. [`auditability-loro-history.md`](./auditability-loro-history.md). -4. **Flutter and desktop bypass the runtime boundary**: both hold a raw `Db` instead of `AtomicNode`, which is how gap 1 happened. [`atomic-lib-runtime.md`](./atomic-lib-runtime.md). -5. **`SYNC_PUSH` is an unsigned cross-trust import** gated by one drive-level ACL verdict; no grant-chain proof exists. [`authorization-sync.md`](./authorization-sync.md). -6. **npm is seven betas behind**: `@tomic/*` still serves `0.41.0-beta.0`. The `npm` release job exists since 2026-09-15; the next tag publishes once `plugin` and `edit-mode` have trusted-publisher entries. [`production-readiness.md`](./production-readiness.md). -7. ~~**No Rust outbox**~~ Ported 2026-09-16 (`lib/src/sync/outbox.rs`), drained over WS and over live Iroh links. [`unified-sync.md`](./unified-sync.md), [`serverless-p2p.md`](./serverless-p2p.md). -8. ~~**Desktop CSP is disabled**~~ Set 2026-09-15, pending a packaged-build smoke test. [`security-audit-2026-09.md`](./security-audit-2026-09.md). -9. **Plugins and code-first schemas live only in unmergeable PRs** (#1307 at 532 files with conflicts, #1262 a stale draft). [`plugins.md`](./plugins.md), [`json-schema-code-first.md`](./json-schema-code-first.md). -10. ~~**Dashboards have no entry point and fork review shows only a count.**~~ Both closed 2026-09-16: a table's dashboard is a view tab, and the fork bar shows the per-property diff. Still open there: suggest-for-non-writers, per-property revert, Canvas forks. [`dashboards.md`](./dashboards.md), [`drafts-and-suggestions.md`](./drafts-and-suggestions.md). +## Biggest gaps (reconciled against code 2026-09-18) + +Ranked by impact. Each links to the plan that owns the work. Gaps closed since +the 2026-09-15 pass (durable Flutter writes, write-endpoint rate limits, +envelope replication, the Rust outbox, desktop CSP, dashboard entry point and +fork review diff) are recorded in their owning plans, not here. + +1. **The query index silently drops rows.** A drive-scoped, sorted, + class-filtered query returns fewer members than exist: the ignored test + `is_a_encodings_all_match_the_class_constraint` in `lib/src/db/test.rs` + reproduces `query_sorted_indexed` returning 2 of 3 matching rows, and + [`silent-failures.md`](./silent-failures.md) carries two open entries for + the same shape (5 of 22 Houseplants rows). PR in flight. +2. **Flutter and desktop bypass the runtime boundary.** Flutter keeps a + `static DB: OnceLock>` and wraps it in `AtomicNode::from_db` per + call (`flutter/rust/src/api/simple/state.rs`); the desktop app holds a raw + `atomic_lib::Db` (`desktop/src/lib.rs`). [`atomic-lib-runtime.md`](./atomic-lib-runtime.md). +3. **`SYNC_PUSH` is an unsigned cross-trust import** gated by one drive-level + verdict (`admit_drive_write` / write rights in `lib/src/sync/engine.rs`); + no `AuthorizationProof` type exists in `lib/src`. [`authorization-sync.md`](./authorization-sync.md). +4. **The store grows with every write and is never compacted.** `sign_at` + (`lib/src/commit.rs`) exports a full Loro snapshot per commit although its + docstring promises an incremental update, and compaction exists only as the + `atomic-server compact` subcommand (`server/src/bin.rs`). PR in flight. + [`disk-storage-and-persistence-optimization.md`](./disk-storage-and-persistence-optimization.md). +5. **npm is seven betas behind**: the workspace is at `0.41.0-beta.7`, the + registry serves `0.41.0-beta.0`. The `npm` job in `release.yml` exists + since 2026-09-15 and #1355 (merged 2026-09-18) fixed its build order and + publish step; the next `v*` tag publishes once the remaining + trusted-publisher entries are in place. [`production-readiness.md`](./production-readiness.md). +6. **The managed-node bootstrap grace has no reaper.** `SyncPolicy` admits + any drive for ten minutes after first sight (`lib/src/commit.rs`, + `lib/src/sync/policy.rs`) and nothing reaps drives that never enroll. + [`cloud-sync-managed-node.md`](./cloud-sync-managed-node.md) item 4, + [`foss-public-host-mode.md`](./foss-public-host-mode.md) Phase 3. +7. **Code-first schemas live only in an unmergeable PR.** `defineSchema` and + frozen `did:ad:` schemas are in #1262, a draft with conflicts last touched + 2026-09-01; nothing of it is on `develop`. The plugin model itself is no + longer in this position: #1500 (merged 2026-09-17) carried it onto + `develop`, and #1307 is now behind `develop`. [`json-schema-code-first.md`](./json-schema-code-first.md), + [`plugins.md`](./plugins.md). ## Active - [Drive sharing and hosting state](drive-sharing-state.md) — verified transition for unenrolled drives; authoritative seat counts and staging acceptance remain. -- [Website publishing](./website-publishing.md) — FOSS publication adapter and shared contract for managed SaaS hosting. - -- [Assistant-authored websites](./assistant-websites.md) — first local prototype validated; plugin abstraction audit and remaining SaaS deployment work. +- [Website publishing](./website-publishing.md) — FOSS publication on `develop` since #1500 (2026-09-17); managed SaaS adapter and the open follow-ups carried from that PR's handoff. +- [Assistant-authored websites](./assistant-websites.md) — prototype merged with #1500; plugin abstraction audit and remaining SaaS deployment work. Remaining work, not "this file exists." @@ -63,7 +89,7 @@ browser flow; standalone recovery remains self-managed. | Document | Status | | --- | --- | -| [`production-readiness.md`](./production-readiness.md) | **Gate list.** What stands between `develop` and production: npm publishing, rate limiting, library-owned durability, desktop CSP, managed-node abuse gate, source maps, SaaS billing checks. | +| [`production-readiness.md`](./production-readiness.md) | **Gate list.** Still open between `develop` and production: the first npm publish on a `v*` tag (job landed, #1355 merged 2026-09-18), green paired server/SaaS CI, the managed-node abuse gate, private source maps, backend synthetic error reporting, SaaS billing checks. Rate limiting, library-owned durability and desktop CSP shipped 2026-09-15. | | [`security-audit-2026-09.md`](./security-audit-2026-09.md) | **Mostly fixed** (beta.6, plus B7 CSP, C18 and rate limits on 2026-09-15). Open: C16 process-global import flags, C17 DID watched-query leak, C20, C24 loopback NFS, permissive CORS, client errors as 500, section F, transitive advisories via actix-http and iroh 0.35. | | [`drive-sharing-state.md`](./drive-sharing-state.md) | **In progress.** Verified transition for unenrolled drives shipped (#1466). Remaining: authoritative per-drive editor usage from the backend, root cause of the retained remote routing, staging acceptance. | | [`cloud-subscription-panel.md`](./cloud-subscription-panel.md) | **Partial.** Profile and link-invite steps shipped. Remaining: SaaS email invitation to drive authorization in one journey, invitation usage limits, paid-seat approval, real seat counts, 50 GB pool. | @@ -86,7 +112,7 @@ browser flow; standalone recovery remains self-managed. | [`serverless-p2p.md`](./serverless-p2p.md) | **Planned.** Device sync without a hub (written same-agent-first; admission is rights-based since 2026-07-17). AUTH-before-SYNC and the `AUTH.requestedSubject`↔drive binding landed 2026-09-01 (Iroh). Live-link destroys travel as signed `COMMIT` frames since 2026-09-03. P0 remaining: require envelopes on every `remove[]` (`Tree::Envelopes` exists; nothing blocks it). `AtomicTransport` / `SyncSession::serve` first slice landed 2026-09-05 with only `ChannelTransport`; outbox port and the four `sync_drive_with_peer*` variants are open. | | [`foss-public-host-mode.md`](./foss-public-host-mode.md) | **Partial.** Phase 1–2 built; OQ5 library path closed 2026-09-05 (`admit_unknown_drive`: Public never creates, Owner enrolls only the owner). Phase 3 (rate limits, Iroh stream refusal) is untouched. | | [`authorization-sync.md`](./authorization-sync.md) | **Draft.** P1 done, P2 partial (`classify_auth_impact` exists). P3 open: no `AuthorizationProof`, signer still auto-inserted into `write`, `genesis_signer()` has no non-test caller. P4 open: `SYNC_PUSH` imports raw Loro on a drive-level verdict. | -| [`unified-data-layer.md`](./unified-data-layer.md) | **Partial.** Browser/JS: one ingress, one outbox, one subscription model. Atomic writes, outbox, ingress entry points and immutable read/save subscriptions shipped. Remaining: consumer migration (38 `addResource` call sites, 42 `CommitBuilder` refs) and boundary extraction. | +| [`unified-data-layer.md`](./unified-data-layer.md) | **Partial.** Browser/JS: one ingress, one outbox, one subscription model. Atomic writes, outbox, ingress entry points and immutable read/save subscriptions shipped. Remaining: boundary extraction and the `CommitBuilder` review (31 non-test references, all in `browser/lib/src`; `addResource(` has no non-test call site left in `browser/data-browser/src`, counted 2026-09-18). | | [`loro-source-of-truth.md`](./loro-source-of-truth.md) | **Partial.** Sparse `datatypes` map + Phase 2a–2c shipped (`Tree::Resources` is a derived cache). Remaining: drop the untagged heuristic and the 3-way `build_state_doc` fallback, snapshot backfill (Phase 4), Phase 1.6 `Value` reshape (~966 sites), Flutter undo. | | [`atomic-lib-runtime.md`](./atomic-lib-runtime.md) | **Partial.** `AtomicNode` is the binding runtime; the WASM `ClientDb` is its only adapter. Flutter and desktop still hold a raw `Db`; the node has no blob API; `NodeConfig` was cut. Library-owned durable flush is the first slice. Local KV FTS landed in [`local-search.md`](./local-search.md). | | [`genesis-self-verifying.md`](./genesis-self-verifying.md) | **Partial.** Server and browser mint and verify inline genesis certs. Remaining: DataRoute verify UI, `genesis` propval immutability, cert-signed `drive` in `check_rights`. | @@ -102,16 +128,16 @@ browser flow; standalone recovery remains self-managed. | [`unified-templates.md`](./unified-templates.md) | **Initial slice shipped** (#1428: catalogue, editable previews, template chat). Remaining: portable format, website adapter, demo-lifecycle extraction, provenance and resume, live-AI acceptance. | | [`dashboards.md`](./dashboards.md) | **First slice shipped**, reachable from its table as a `dashboard` view tab since 2026-09-16. Open: the set-level action verb, parameters, templates shipping a dashboard. | | [`content-i18n.md`](./content-i18n.md) | **LocalizedText + template locales shipped.** Nothing in the app resolves translation siblings. Remaining: TranslationsBar, `useTranslation`, `/query` `lang`, search language filter. | -| [`website-templates.md`](./website-templates.md) | Template repair complete (DID), two-locale E2E exists. Its CMS list is the website view of `drafts-and-suggestions.md` and `content-i18n.md`. Open: publication visibility, CMS origin, in-page edit affordance, canonical paths. PRs #1498 and #1500 (assistant-designed and self-hosted sites) are in flight. | +| [`website-templates.md`](./website-templates.md) | Template repair complete (DID), two-locale E2E exists. Its CMS list is the website view of `drafts-and-suggestions.md` and `content-i18n.md`. Open: publication visibility, CMS origin, in-page edit affordance, canonical paths. Self-hosted publishing (#1500) merged 2026-09-17; the assistant-designed-sites prototype PR #1498 is still an open draft against `feat/plugin-model`. | | [`structural-problems-index.md`](./structural-problems-index.md) | **Live index.** React subscription audit is partial; save-state APIs shipped with two consumers. Browser metadata cleanup and subject-brand consumers remain; server subscription work is complete. | | [`react-compiler-resource-proxy.md`](./react-compiler-resource-proxy.md) | **Partial.** Immutable read/save status hooks and the data-inspector subscription shipped; about 13% of property reads are still render-time getters, migrating one regression at a time. | | [`canvas-undo-consolidation.md`](./canvas-undo-consolidation.md) | Phase A + C landed (browser). Phase B (Flutter action-stack removal, `replace_list_items`) open and unstarted. | | [`index-performance.md`](./index-performance.md) | First tranche shipped. Remaining: exact counts, typed sort keys, batched reads, watched-filter LRU, re-profiling. Structural permission-check fix is `zones.md`, not built. | -| [`disk-storage-and-persistence-optimization.md`](./disk-storage-and-persistence-optimization.md) | **Proposal.** Full-snapshot writes per commit, no auto-compaction, O(file) open fsync, SIGTERM path. Fresh e2e data dirs are done. | +| [`disk-storage-and-persistence-optimization.md`](./disk-storage-and-persistence-optimization.md) | **Proposal, PR in flight** (gap 4). Full-snapshot writes per commit, compaction only via the CLI, O(file) open fsync, SIGTERM path. Fresh e2e data dirs are done. | | [`virtual-drive.md`](./virtual-drive.md) | **Shipped** as a local NFS mount in the Tauri desktop app (`desktop/src/vfs.rs`, unauthenticated loopback, audit C24). Still proposal: headless-server mount, FUSE/WinFSP, native cloud-sync APIs, mobile providers. | | [`commit-retention-and-state-certificates.md`](./commit-retention-and-state-certificates.md) | **Superseded.** Envelope-on-resource replaced the retention design; Phase 1 and 2.5 shipped. Only the `stateHash` certificate and per-resource `retention` propval remain from this document. | | [`s3-blob-storage.md`](./s3-blob-storage.md) | **Partial.** Server-wide S3, verified migration and SaaS enforcement implemented. The backend trait is `get/put/size` only: streaming, delete/GC, per-tenant configuration and encrypted Vault attachments remain. | -| [`plugins.md`](./plugins.md) | **Partial, off `develop`** — one plugin model (`run` end to end, per-app agents, unattended runs). The code lives on `feat/plugin-model` (PR #1307, 532 files, conflicts with `develop`) and #1482. On `develop` the plugin RPC still answers "not implemented". | +| [`plugins.md`](./plugins.md) | **Partial, on `develop` since #1500** (2026-09-17) — one plugin model (`run` end to end, per-app agents, unattended runs): `browser/plugin`, `browser/lib/src/plugin-run.ts`, `integrations/`, the `wasm-plugins` server routes. PR #1307 is still open but behind `develop` (one file only on that branch). Remaining work is the list in #1307 and `extension-architecture.md`. | | [`json-schema-code-first.md`](./json-schema-code-first.md) | **Proposal**; nothing on `develop`. `defineSchema` + frozen `did:ad:` schemas are in PR #1262, a draft last touched 2026-09-01. | | [`android-data-reuse.md`](./android-data-reuse.md) | **Draft.** One store/agent/Iroh node per Android device. Nothing built. Supersedes `on-device-atomic-daemon.md`. | | [`SDK-API-design.md`](./SDK-API-design.md) | SDK / agent DX direction. | @@ -144,8 +170,8 @@ Not top-level plans. Indexed so they do not go missing. | [`arc-actor-message-payloads.md`](./arc-actor-message-payloads.md) | **Stretch shipped.** `SendFrame { Arc<[u8]> }` encode-once is in; only the `atomic_lib` `loro_update: Vec` question remains and is low value now. | | [`sync-onboarding-ux.md`](./sync-onboarding-ux.md) | Reference. Cross-client copy for what can reach what. Companion to [`device-pairing.md`](./device-pairing.md). | | [`main-drive-and-paths.md`](./main-drive-and-paths.md) | Strategy. DID-branch deployment: root drive, legacy URLs, human-readable paths. Phases 1–3 unstarted. | -| [`actions.md`](./actions.md) | **Steps 1–4 shipped.** Registry drives ⌘M, ⌘K (capped prefix match), hotkeys, the shortcuts overlay/page, and simple AI tools. Remaining: MCP projection when a server exists (PR #1347 is the plan). | -| [`silent-failures.md`](./silent-failures.md) | Living log of error-handling failures that reported success (2026-08-21). Carries M8 from the pairing field test and the unreproduced Safari fork query. | +| [`actions.md`](./actions.md) | **Steps 1–4 shipped.** Registry drives ⌘M, ⌘K (capped prefix match), hotkeys, the shortcuts overlay/page, and simple AI tools. Remaining: MCP projection when a server exists ([`mcp-endpoint.md`](./mcp-endpoint.md), #1347 merged 2026-09-18). | +| [`silent-failures.md`](./silent-failures.md) | Living log of error-handling failures that reported success (2026-08-21). Carries the two open query-index entries (gap 1), M8 from the pairing field test, the unreproduced Safari fork query and the e2e harness traps from the #1500 pass. | Closed decisions, as-built records, closed explorations and fixed notes live in [`completed/`](./completed/): the five decisions above, the 2026-07 sync diff --git a/planning/WEBSITE_HANDOFF.md b/planning/WEBSITE_HANDOFF.md deleted file mode 100644 index 7b458af6bc..0000000000 --- a/planning/WEBSITE_HANDOFF.md +++ /dev/null @@ -1,332 +0,0 @@ -# Atomic websites: implementation handoff - -Updated 15 September 2026. Read this before continuing work. This is a working -prototype with focused tests, not a production-ready hosting product. - -## Start here - -- Repository: `ontola/atomic-server`, checkout `/private/tmp/atomic-pr-website` - (`/tmp/atomic-pr-website` is the same location). -- Branch: `codex/website-self-hosted-publishing`. -- Draft PR: https://github.com/ontola/atomic-server/pull/1500 -- PR base: `codex/assistant-website-publication`, **not develop**. This is stacked - on the Assistant website work originating from the plugin work. Inspect the - current dependency/merge state before changing its base. -- Last implementation commit: `cb3e6ae00` (pushed). The handoff itself follows it. -- SaaS checkout: `/Users/joep/dev/atomic-saas`. The implemented hosting backend in - this PR is in atomic-server; a managed SaaS publishing adapter is still needed. -- Read repository AGENTS.md files and `browser/data-browser/UI_COMPONENTS.md` - before editing. Check git status and preserve existing work. -- Companion architecture document: [website-publishing.md](website-publishing.md). - This handoff supersedes its stale references to a custom “Publishing options” - menu: website actions now use the standard resource More menu. - -Do not reset the user's database or identity, overwrite another checkout, or -restart unrelated servers. Create fresh test resources for reproduction. Never -copy invitation tokens, signing secrets or credentials into docs/logs. - -## Product intent and decisions - -The user opens Atomic and asks, “I want a website for my bakery.” The Assistant -finds existing Products, prices, documents and photos, then creates/designs the -website using those resources. Atomic remains the source of truth; the website -must not become a disconnected copy of editable content. - -The user edits through normal tables/documents or inline in the preview. Those -edits save to the original Atomic resources. **Production changes only after -Publish site / Update site.** This explicit publication boundary is intentional, -including after inline editing. Rollback changes delivered website content, not -source records, operational app data or form responses. - -UX requirements from repeated user feedback: - -- One blue primary button at a time: Publish site or Update site. One action - performs snapshot preparation, upload and activation; no mandatory manual - release/upload/review sequence. -- Hosting status refreshes automatically. No “Refresh hosting status” button. -- “Edit with AI” is now back next to “Edit on page” in the preview toolbar as a - subtle button. Both that shortcut and the More action share the same callback. - The More action currently retains the label “Design with AI”. -- View site, versions, unpublish and export operations use standard context - actions. Page-specific actions precede generic actions with one divider. -- Normal View/Data View are ordinary icon actions; hide the active view action. -- Use existing ResourceRow/components for content, meaningful labels and types. - Add content is a button. Avoid a custom CMS UI full of technical export terms. -- Do not reload the preview merely because AI/sidebar state changes. -- Do not add version resources to the sidebar or version-specific show-more UI. -- Show failures through `store.notifyError`, with a useful visible failure state; - don't leave a spinner forever or silently disable the only useful action. - -Longer-term intent: strong interactive apps as well as beautiful websites, reuse -the plugin abstractions and CMS editing controls, FOSS hosting plus managed SaaS. -**The current implementation is static-v1; that wider ambition is not delivered.** -CLI uploads are not the main authoring flow. Registrar/DNS purchasing is deferred. -Account entitlements should eventually be visible in Atomic, with actual payment -in the trusted SaaS account flow. No approved public pricing or automatic overages. - -## What is implemented - -### Authoring and preview - -`browser/data-browser/src/chunks/Website/` contains the implementation: - -| File | Responsibility | -| --- | --- | -| `websiteModel.ts`, `websiteTools.ts` | Website resource/config and Assistant authoring tools | -| `websiteExport.ts`, `renderWebsite.ts` | Selected source snapshot and static rendering | -| `WebsitePage.tsx` | Content sidebar, draft preview, AI shortcut, export context actions | -| `WebsitePreview.tsx`, `useWebsitePreviewHtml.ts` | Preview rendering and media URLs | -| `WebsiteInlinePreview.tsx`, `websiteInlineEditing.ts` | Authorized inline source editing | -| `WebsiteHosting.tsx`, `hostingClient.ts` | Status, publish/update, versions, rollback and HTTP client | -| `WebsiteExportPage.tsx` | Frozen version preview, including legacy exports | -| `websiteMedia.ts`, `websiteAssets.ts`, `optimizeWebsiteImage.ts` | Media selection, blobs and optimization | - -Preview subscriptions filter changes to website design rather than rebuilding on -any resource notification. Source changes still trigger debounced rebuilds. The -previous preview remains visible while refreshing. Unchanged output preserves -iframe identity. This is a correctness improvement, not a measured performance SLA. - -Inline editing supports selected string, integer and float table fields. Numbers -are parsed as numbers, reject invalid/nonfinite values, and integers require safe -whole values. Commits check current permissions, binding membership and conflicts. -Document sections mount the existing `CollaborativeEditor` in a React portal in -the preview iframe, reusing structured rich text and normal autosave. - -The latest fix passes the iframe body as `menuContainer` to the editor. Resource -and slash suggestion menus are adopted into that document before positioning; -keyboard scrolling uses its ownerDocument. Embedded editor backgrounds are -transparent. Styled-components styles target the iframe head. The preview remains -sandboxed without script permission; trusted Atomic code owns the editor/writes. - -### Versions and delivery - -- Browser uploads a version-1 WebsitePackage: code files, asset references and - optional **private** metadata for reconstructing previews. -- Server stores code and a version-2 manifest as immutable BLAKE3 blobs through - BlobBackend (local or configured internal S3). Images are separate HTTP blobs, - never base64/bytes embedded in HTML. S3 stays private behind serving endpoints. -- KV/redb stores membership, deployment IDs/timestamps, active pointer and history. - New exports create no Atomic child resources and no website-release pointer. - Legacy resource/KV exports remain readable; they are not deleted. -- Identical exports deduplicate. The current pilot caps distinct deployments at - 20 and activation history at 100. There is no blob garbage collector. -- Publish uses expectedRevision; stale activation gets HTTP 409. Failed upload or - build preserves the existing public site. Upload alone does not publish. -- Public files are served on a separate customer origin. Delivery does not read - the authoring drive. HTML pins runtime/assets to its version. -- Previously activated versions remain accessible while the site is published; - unpublishing hides them all. Downloaded copies cannot be recalled. -- Source edit authority is narrower than publication: current publishing requires - write access to the website and containing drive root. - -Server entry points: `lib/src/website.rs`, `lib/src/db/website.rs`, -`server/src/handlers/website.rs`, `server/src/blob_storage.rs`. - -Control API (signed requests): - -- GET `/website-hosting?project=...&drive=...`: status/history/URL. -- POST `/website-hosting/deployments?project=...&drive=...`: private package upload. -- GET `/website-hosting/preview/{deployment}?project=...&drive=...`: private JSON - package, no-store; never serve customer HTML on the editor origin. -- POST `/website-hosting/activate?project=...&drive=...`: - `{ expectedRevision, deployment }`; null unpublishes; an old ID rolls back. - -### Media - -Browser optimization preserves originals and generates derivatives with longest -edge at most 1920px, targeting 600 KB WebP. Small images and GIF animation are -preserved; large GIFs have an explicit rejection. Input caps are 50 MB/80MP. -Errors identify the source/field/page. Private unpublished asset availability is -currently tied to local cached blobs; see improvements below. - -## Improvements needed, in suggested order - -### 1. Finish validating inline RTE behavior (immediate) - -The user reported @/# appearing frozen and a white editor background. The last -commit fixes the known menu-document mismatch and background, and focused tests -pass. **Do not interpret this as all rich-text features verified.** - -Next reproduce in Firefox (the user's reported browser) and a real bakery page. -Test selecting a mention, arrow navigation, escaping then reopening, formatting -bubble menus, link/color popovers, image insertion/upload, tables, drag handles, -undo/redo, selection across blocks, and long/scrolled documents. Some shared RTE -components still refer to global document/window or use portal defaults. Audit -these against iframe ownership rather than loosening iframe sandbox permissions. -Check CSS/font inheritance, dark backgrounds and popup readability. `# ` with a -space is the heading input rule; a bare # is not established as a suggestion menu. - -Acceptance: each supported control works visibly in the iframe, saves to the -original resource, survives reload and does not leak keys/credentials to site code. -Add focused regressions for actual failures; don't build a separate rich-text editor. - -### 2. Preview speed, invalidation and actionable failures - -The user repeatedly reported slow “Preparing preview”, unnecessary reloads and -missing content errors. Some fixes exist; investigate remaining bottlenecks with -measurements (resource fetching, image decode/optimization, render, blob URL setup). -Avoid rebuilding/re-fetching every asset on unrelated changes. Cover remote source -updates, additions/removals in tables, replacing files, concurrent inline edits and -navigation. Preserve cursor/selection during relevant background updates. - -There is an Up to date / Unpublished changes state. Audit that it compares actual -published output to current draft across related rows/documents/assets; never -infer “clean” from an unavailable source or failed preview. Explain disabled -publish states. Missing resource errors should offer practical recovery (repair or -remove the selected binding) rather than leave a long DID and a dead end. Do not -silently publish with omitted private/unreadable content. - -### 3. Managed atomic-saas publishing (major missing deliverable) - -Implement a hosting destination adapter with the same status/upload/private-preview/ -activate semantics. Reuse the builder and package contract, not a second CMS. -SaaS must validate account-to-drive ownership, hosting entitlement and publishing -rights; use the trusted account session flow without forwarding signing secrets. -Use durable transactional compare-and-swap for activation, private object storage -for immutable blobs, and a customer-content serving origin independent of a -managed drive process. FOSS redb locking is not a multi-process SaaS transaction. - -Prove staging publication, private previews, unauthorized access denial, activation -conflict, failed-build preservation, rollback, unpublish, source drive offline, -and image delivery. Live S3/managed delivery has NOT been demonstrated here. -Coordinate atomic-server dependency pin updates only with compatibility checks. -Keep shared drive subscriptions from being charged again for every website. - -### 4. Storage/delivery production work - -Add retention and reference-aware GC (including failed uploads and concurrent -publication), quota enforcement/reporting, meaningful cap-reached UX, backups and -restore tests covering BOTH metadata and blobs. Measure cache behavior and serving -cost/latency. Never delete a blob still referenced by an active or retained version. -Verify internal S3 end to end, not merely the local BlobBackend path. Make unpublished -exports/assets portable across devices; currently local cache alone is insufficient. -Document manifest compatibility/migration and limits before treating this as stable. - -### 5. Reassess renderer/plugin abstractions for the actual product - -Current static rendering does not prove arbitrary AI designs or powerful interactive -apps. Review the plugin package, grants and bridge abstractions against design -freedom, structured bindings, reusable CMS controls and a standalone app host. -Prefer extending shared abstractions; don't create parallel form schemas or auth. -Current static CSP blocks arbitrary scripts, live network interactions/workers and -form actions. Changing that requires an explicit runtime/security design, not just -turning off restrictions to make a demo work. - -Forms should reuse the Forms implementation associated with PR #1281 once a -compatible revision is verified; its current merge status was not rechecked here. -Keep site deployment, form availability and private response rows independent. -Website rollback must not reopen forms or delete responses. Preview submissions -need an explicit test path. Domain registration/DNS and arbitrary runtimes remain -out of this immediate milestone. - -### 6. Remaining authoring/product polish - -Review document File embeds, a folder-to-gallery flow, labels/icons, responsive -layout and accessibility. Audit screenshot capture and Assistant input drag/drop -requests against current code before assuming completion; they were discussed but -are not established as delivered by this handoff. Likewise, prior chat persistence, -rate-limit/tool-result recovery and sidebar chat requests are separate work: inspect -their PRs before conflating them with website-hosting completion. - -## Local development and verification - -These are the previous session's local service paths; check listeners/process -ownership before using/restarting them. Do not replace user state. - -- Vite: `http://localhost:6763`, config `/private/tmp/hosting-vite.config.mts`, - cache `/private/tmp/hosting-vite-cache`, log `/private/tmp/hosting-vite.log`. - Aliases use this checkout's `browser/lib/dist/index.js` and `browser/react/src/index.ts`. -- Backend: `http://localhost:9897`, binary - `/private/tmp/atomic-website-rust-build/debug/atomic-server`. - Data/config/cache under `/private/tmp/atomic-hosting-node/{data,config,cache}`. -- Website origin: `ATOMIC_WEBSITE_ORIGIN=http://sites.localhost:9897`. - Public URLs are `http://.sites.localhost:9897/`. -- Do not restart the backend for UI-only changes. If Rust changes, ensure the - running binary actually includes them before claiming live validation. -- Vite/Wuchale updates locale .po files. Inspect those diffs; avoid extraction and - resets racing each other. HMR can disturb tests if edits continue during a run. - -From the atomic-server checkout: - -```sh -browser/node_modules/.bin/oxfmt -c browser/.oxfmtrc.json -browser/node_modules/.bin/oxlint -c browser/.oxlintrc.json -``` - -From `browser/data-browser`: - -```sh -../node_modules/.bin/tsc --noEmit -p tsconfig.json -``` - -Focused live browser checks (existing services required): - -```sh -FRONTEND_URL=http://localhost:6763 SERVER_URL=http://localhost:9897 WEBSITE_HOSTING_E2E=1 \ - browser/e2e/node_modules/.bin/playwright test \ - --config browser/e2e/playwright.config.ts \ - browser/e2e/tests/website-inline-content.spec.ts \ - browser/e2e/tests/website-preview-stability.spec.ts \ - --project chromium --workers 1 --trace off --reporter line -``` - -Other suites in `browser/e2e/tests`: `website.spec.ts`, `website-publishing.spec.ts`, -`website-media.spec.ts`, `website-errors.spec.ts`, `website-versions.spec.ts`, -`website-export.spec.ts`. Use the normal isolated E2E setup if existing services -aren't available; don't point destructive tests at the user's drive. - -### Evidence and limits - -At cb3e6ae00: the two focused Chromium tests above passed together (18.3s), -TypeScript passed, targeted lint had no errors (existing set-state-in-effect -warning in CollaborativeEditor). Tests cover visible slash/@ menus, heading -input, transparent editor background, numeric/document save and reload, and -AI opening without iframe replacement. They do not cover every RTE feature. - -Earlier commits passed focused publication/authoring/media/error/version browser -checks and unit/Rust tests. They were not all rerun after the final commit. -No claim of green full CI, Firefox parity, production deployment, load testing or -live S3 verification. Read current PR checks before merge. - -#### Session of 15 September 2026 (after f63987493) - -- `website-inline-rte.spec.ts` (new, fixture in `website-inline-fixture.ts`) - exercises bold, node select, the link popover, undo/redo, `@` mention - navigation/escape/reopen, slash escape/reopen, image picker, drag handle and - reload persistence inside the preview iframe. It passed in Firefox (2/2, via a - scratch config adding a Firefox project) and in Chromium, but see the flake below. -- Fixed: the link popover rendered in the app document, unstyled and at - iframe-relative coordinates (the "frozen" symptom). It now portals into the - bubble menu, which stays shown while the form is open (`BubbleMenu.tsx`, - `PopoverContainer` in `WebsiteInlinePreview.tsx`). The `Set link` button has a - title. Draft rebuilds are paused while inline editing (`WebsitePage.tsx`). -- Not fixed, pre-existing: characters intermittently vanish or land in a new - paragraph while typing in the iframe editor at human speed (40ms per key). - Reproduced with all session changes stashed (`website-inline-content.spec.ts` - failed 2/3). The main document editor also commits roughly once per keystroke - at that speed (measured 35 commits for 43 characters), so the save scheduler - is a suspect; `loro-prosemirror` logs "Cannot find the loroNode" around the - failures. Treat both inline specs as flaky in Chromium until this is found. -- Firefox at 0ms synthetic key delay mis-sequences keys (not a product bug); - adding `white-space: pre-wrap` to the editor made Chromium worse and was reverted. -- Website page: AI edit / Page edit buttons moved into the header row with - icons, "Content" and "Live draft preview" labels removed, preview fills the page. -- AI chats: titles are generated on the first user message (fallback on the - reply) and include an emoji shown in the sidebar and the chat panel heading. -- Chat loading: messages load in two rounds instead of one per message, and - `store.getResource` / the new `store.getResources` are local-first (embedded - database before the server). A `GET_MANY` (0x15) / `GET_MANY_RESULT` (0x16) - WebSocket frame pair, capability `get-many`, batches server fetches; on - localhost it measured within noise of pipelined single GETs (about 200ms - for 120 subjects either way), so the transport was not the bottleneck. - -## Suggested next session - -1. Confirm branch/status, read applicable repo instructions, and inspect PR checks. -2. Reproduce the user's inline RTE interactions in Firefox and address any remaining - ownerDocument/portal issues. Keep the working publication and source-save paths. -3. Measure preview work and fix one concrete bottleneck or stale-content case. -4. Implement the managed SaaS adapter as a separately reviewable milestone, with - coordinated compatibility tests and a real staging demo. -5. Report exactly what is verified, what remains a prototype, and which user-visible - behavior to test next. Do not substitute a toast/build for the requested outcome. diff --git a/planning/assistant-websites.md b/planning/assistant-websites.md index 89b31a4c66..78535dd7b3 100644 --- a/planning/assistant-websites.md +++ b/planning/assistant-websites.md @@ -2,9 +2,11 @@ # Assistant-authored websites: first implementation and abstraction review -Status: first local prototype validated on `codex/assistant-website-publication`, based -on `feat/plugin-model` at `d0e49b8b9`. The source worktree's staged and unstaged -changes were preserved. No production deployment or domain integration. +Status: prototype built on `codex/assistant-website-publication` (PR #1498, still +an open draft against `feat/plugin-model`) and merged into `develop` with the +self-hosted publishing PR #1500 that was stacked on it (2026-09-17). The +validation notes below describe the prototype as it was validated on that +branch. No production deployment or domain integration. ## Product decisions from the discussion diff --git a/planning/disk-storage-and-persistence-optimization.md b/planning/disk-storage-and-persistence-optimization.md index aa5ea75ea4..7d85c3a604 100644 --- a/planning/disk-storage-and-persistence-optimization.md +++ b/planning/disk-storage-and-persistence-optimization.md @@ -1,6 +1,7 @@ # Disk Storage & Persistence Optimization -> **Status:** Proposal (2026-06-04). Diagnoses why server boot time and overall +> **Status:** Proposal (2026-06-04); fix 1 and fix 2 have a PR in flight +> (2026-09-18). Diagnoses why server boot time and overall > performance degrade as the store grows, and proposes fixes. Builds on > [`commit-retention-and-state-certificates.md`](./commit-retention-and-state-certificates.md) > (history retention is node policy) and @@ -64,10 +65,11 @@ scales with the file, for two reasons the code already documents: Three compounding behaviors, all in code: 1. **Full Loro snapshots on every commit — not deltas.** `sign_at` - (`lib/src/commit.rs:1121`) calls `doc.export_snapshot()`, serializing the + (`lib/src/commit.rs`, `fn sign_at` near line 1493 on 2026-09-18) calls + `doc.export_snapshot()`, serializing the resource's **entire** CRDT document after applying the change. A 1-character title edit re-stores the whole resource's Loro state. **NB:** the docstring - at `commit.rs:1088` says *"an incremental update is exported"*, but the call + just above it says *"an incremental update is exported"*, but the call is `export_snapshot()` (full). This mismatch looks **unintended** and is the single biggest growth lever. 2. **History retention.** Each commit is retained (the `previousCommit` audit @@ -106,7 +108,7 @@ what the `commit.rs:1088` docstring already intended. - The materialized "current state" snapshot per resource (`TABLE_LORO_SNAPSHOTS`) can stay full for fast reads; it's the **per-commit `loroUpdate`** that should be a delta — that's the part multiplied by history. -- **First step:** confirm `export_snapshot()` at `commit.rs:1121` is an +- **First step:** confirm `export_snapshot()` in `sign_at` is an unintended full-write (vs. a deliberate choice for import-merge correctness), and that `from` (the prior version vector) is available at sign time. - **Downstream wins:** the per-commit `loroUpdate` is also the payload that gets @@ -161,7 +163,7 @@ change, but it's the fastest path to a non-flaky suite and isolates real bugs ## Open questions -- Is `export_snapshot()` at `commit.rs:1121` load-bearing for import-merge +- Is `export_snapshot()` in `sign_at` load-bearing for import-merge correctness, or safe to switch to an incremental export? (Determines fix #1.) - What's the dead-page ratio of a real aged store? Run `atomic-server compact` on a production-like store and compare before/after size to size the win. @@ -174,7 +176,7 @@ change, but it's the fastest path to a non-flaky suite and isolates real bugs - `lib/src/db/redb_store.rs:93-160` — open path, fsync/repair cost notes, `compact_file`, `set_quick_repair`. -- `lib/src/commit.rs:1085-1124` — `sign_at`; `export_snapshot()` at line 1121 - (full snapshot despite "incremental" docstring at 1088). -- `server/src/bin.rs:119` — manual `compact` subcommand wiring. +- `lib/src/commit.rs` — `sign_at` (about line 1490 on 2026-09-18); + `export_snapshot()` at its end, full snapshot despite the "incremental" docstring. +- `server/src/bin.rs` — manual `compact` subcommand wiring (`Command::Compact`). - `TABLE_LORO_SNAPSHOTS`, `TABLE_BLOBS` — per-resource snapshot + blob storage. diff --git a/planning/e2e-concurrency.md b/planning/e2e-concurrency.md index 0df1baefd7..675ca02b39 100644 --- a/planning/e2e-concurrency.md +++ b/planning/e2e-concurrency.md @@ -388,3 +388,18 @@ at the selected position. Use the existing waitForGridMounted after reload befor selecting the row. Keyboard assertions and timeouts are unchanged. E2E typecheck passes; focused browser validation and the complete comparison remain pending. Trace: Mancave /tmp/e2e-1465-shift-enter-trace.zip. + +## Local worker count versus CI (from the #1500 pass, 2026-09-16/17) + +CI runs **two workers per shard, four shards on Mancave, each shard with its +own `atomic-server`**, with retries (`.dagger/src/index.ts`, `HOST_PROFILES`). +A local four-worker run against one server is much harsher than that and +invents failures: on the #1500 branch `documents`, `username-live` and two +`apps` tests failed at four workers and passed at two, and a rotating pair of +specs (`plugins:909`, `tables:482`, `drive-catalog`, `offline-tables`, ...) +failed purely from load, different each run. Measure locally with +`PLAYWRIGHT_WORKERS=2 PLAYWRIGHT_RETRIES=1` and re-run any red in isolation +before treating it as real. The full-suite numbers on that branch moved from +41 failed to 5 failed / 4 flaky in a day, and roughly half of the original +list was environment (no server running, wrong dependencies, desynced i18n +catalogs, a 1.1 GB data dir) rather than product. diff --git a/planning/e2e-diagnostic-hygiene.md b/planning/e2e-diagnostic-hygiene.md index d19ad7e681..dc49a72262 100644 --- a/planning/e2e-diagnostic-hygiene.md +++ b/planning/e2e-diagnostic-hygiene.md @@ -152,3 +152,36 @@ Fixed statement spacing, shadowed test variables and the database-handoff mock's Validation after cleanup: 381 library tests, nine diagnostic-fixture self-tests, the database-handoff regression and the app typecheck pass. Generated local `dist-hosted` output was moved outside the source tree before linting. + +## Open after the #1500 pass (2026-09-17) + +Carried from that PR's e2e handoff when it was deleted. Fixed on the way and +worth knowing about: a spec can declare a diagnostic that only some topologies +produce (`expect(..., { optional: true })` in `diagnostic-collector.ts`, +because `recovery-option`'s service-worker warning exists only in a production +build); the app no longer fetches the server root for an empty subject +(`normalizeSubject('')` resolved to `http://host/`, which 404s on a DID-drive +server and logged two console errors per render — that was the whole +"diagnostics group" of failures); `useChildren` re-reads a page older than its +own count and a query whose answer predates a new child. + +- [ ] `website-inline-rte` / `website-inline-content` flake: see + [`website-publishing.md`](./website-publishing.md) Remaining work 1. +- [ ] `table-create-perf` (3) and `opfs-init-perf` fail against a Vite dev + server because the `clientdb.*` perf marks are absent from the dev module + graph; a production build made locally failed them differently + (`/app/dev-drive` never reached a drive), so they are unexplained rather + than explained. `recovery-option` is dev-topology only and now declares + its diagnostic as optional. +- [ ] `vault-backup-restore` skips without the atomic-saas control plane on + :3030 (`vault-stack.sh`); it has not been run green against this + branch's own portal. +- [ ] `useDriveClass` (`chunks/PluginRuns/runScript.ts`) subscribes to the + drive's `defaultOntology` only if it exists when the hook mounts; a drive + that gains its first plugin schema in this tab does not re-resolve, so a + freshly created app renders the generic resource page until a reload. A + fix (also subscribe to the drive resource) was written and reverted + because it needs a test that proves it; the `apps` specs pass since the + schema-term lookup was batched, so the gap is no longer test-visible. +- [ ] `second-device-load @smoke` went from about 1 pass in 3 to 6 in 7 after + the `useChildren` fix; the remaining miss is not diagnosed. diff --git a/planning/extension-architecture.md b/planning/extension-architecture.md index 554406b470..6dc43e50e2 100644 --- a/planning/extension-architecture.md +++ b/planning/extension-architecture.md @@ -2,7 +2,8 @@ Status: product direction agreed with the user, 2026-09-08; technical migration in progress. This describes the target, not guarantees already implemented. Current -extension work is on PR #1307; Reflector integration is proposed in PR #1383. +extension code reached `develop` with #1500 (2026-09-17); PR #1307 remains open +for its remaining-work list. Reflector integration is proposed in PR #1383. Workspace navigation, shared view protocol and installation identity have implementation checkpoints below; the complete package and permission migration remains open. diff --git a/planning/plugins.md b/planning/plugins.md index d982062d17..14adafa3ff 100644 --- a/planning/plugins.md +++ b/planning/plugins.md @@ -5,7 +5,7 @@ defines the target boundaries and phased retirement of overlapping models. This document retains earlier design and implementation history; its “one model” title does not mean the existing UI bridges and lifecycles have already converged. -> **Status:** Partial, off `develop`. Track A built 2026-08-21 (`run` end to end: sandbox, planner, applier, secrets, server placement, scheduled and query-triggered runs, auto-apply); Track B revised 2026-08-22 onto that substrate. The code lives on `feat/plugin-model` (PR #1307). Absorbed `llm-wasm-gui-plugins.md`, `importers.md` and `habits-app.md` on 2026-09-01. +> **Status:** Partial, on `develop` since #1500 (2026-09-17), which was stacked on `feat/plugin-model` and carried it in; PR #1307 is still open but now behind `develop`. Track A built 2026-08-21 (`run` end to end: sandbox, planner, applier, secrets, server placement, scheduled and query-triggered runs, auto-apply); Track B revised 2026-08-22 onto that substrate. Absorbed `llm-wasm-gui-plugins.md`, `importers.md` and `habits-app.md` on 2026-09-01. ## Status diff --git a/planning/production-readiness.md b/planning/production-readiness.md index 3dfcf0f108..f3d6c62fe6 100644 --- a/planning/production-readiness.md +++ b/planning/production-readiness.md @@ -9,7 +9,9 @@ owning plan where one exists. Verification logs stay out of this file. - [ ] **Publish `@tomic/*` to npm on a `v*` tag.** The registry still serves `0.41.0-beta.0` (2025-06) for `lib`, `react`, `svelte` and `cli`; `plugin` and `edit-mode` were never published; seven betas have shipped - since. The `npm` job now exists in `release.yml` (2026-09-15). Remaining: + since (the workspace is at `0.41.0-beta.7`). The `npm` job exists in + `release.yml` since 2026-09-15; #1355 (merged 2026-09-18) fixed its + build order (`@tomic/lib` first) and publishes with `--ignore-scripts`. Remaining: trusted-publisher entries on npmjs.com for `@tomic/plugin` and `@tomic/edit-mode` (the other five are configured), then the next tag. - [ ] Green full CI on the release commit including the atomic-saas @@ -25,7 +27,7 @@ owning plan where one exists. Verification logs stay out of this file. - [x] Library-owned durable flush so the Flutter binding stops losing writes on app kill ([`atomic-lib-runtime.md`](./atomic-lib-runtime.md), 2026-09-15). - [x] Desktop CSP (audit B7) set 2026-09-15; needs one packaged-build smoke test before a release. -- [x] npm publish job in `release.yml` (2026-09-15, ported from #1355). First tag after merge publishes seven `@tomic/*` packages through Trusted Publishing; `plugin` and `edit-mode` need their npmjs.com trusted-publisher entries first. +- [x] npm publish job in `release.yml` (2026-09-15, ported from #1355; #1355 itself merged 2026-09-18). First tag after merge publishes seven `@tomic/*` packages through Trusted Publishing; `plugin` and `edit-mode` need their npmjs.com trusted-publisher entries first. - [ ] Managed-node paid-abuse gate: the bootstrap grace admits any drive for ten minutes with no reaper ([`cloud-sync-managed-node.md`](./cloud-sync-managed-node.md) item 4). diff --git a/planning/silent-failures.md b/planning/silent-failures.md index 15ea998ac0..d5e01e945f 100644 --- a/planning/silent-failures.md +++ b/planning/silent-failures.md @@ -81,6 +81,10 @@ away. *Should have:* wiped it. A warning is the wrong shape when the consequence is a failure list that changes every run. *Fixed 2026-08-18 (87e12c9a) — wipes by default, `--keep-store` to override.* +The same shape returned on the #1500 branch with a custom data dir that +`e2e-server.sh` does not manage: at 1.1 GB, all six `apps` tests and most of +`plugins` failed a 10s `iframe[title="App"]` wait; a fresh data dir took the +two specs from 16 failures to 1. Wipe before any full run. **`waitForSearchIndex(page)` with no query is a 1.5s sleep wearing a readiness-helper's name.** @@ -94,6 +98,61 @@ a name that promises a readiness check. Passing two spec files with two `-g` filters ran one test and reported `1 passed`. Nothing indicated the other filter had been discarded. +**redb's lock outlives the listening socket, so a restart that waits for the +port runs the suite against nothing.** *(open)* +Killing the server frees the port within a second, but `atomic.redb` stays +locked for several more. A restart that only waits for the port exits at once +with `Failed to create redb ...: Database already open. Cannot acquire lock.`, +and every spec then fails in ways that look like product defects: missing menu +items, disabled buttons, empty panels. On the #1500 branch this accounted for +most of one session's "unexplained" failures. `e2e-server.sh` refuses a busy +port before starting, but nothing in the repo waits for HTTP 200 or retries the +spawn. +*Should have:* gate readiness on `curl` returning 200, never on the port, and +retry the spawn (not just the wait) when the log says "Database already open". +Give a fresh server about ten seconds before the first spec; within that +window `menu-item-website-prepare` renders but stays disabled. + +**Compiled i18n catalogs silently shift every string.** +The `wuchale` plugin rewrites `src/locales/*.po` while Vite runs; the compiled +catalogs live in `src/locales/.wuchale/`, which is gitignored. So +`git checkout -- src/locales/` restores the `.po` files and leaves compiled +artifacts from some other state, and every indexed string lookup shifts. +Missing entries render as `[i18n-404:NNN]`, which is visible; *shifted* entries +render as a different real string, which is not. That is how +`iframe[title="Website preview"]` became a Google Calendar sentence and every +website spec failed, an hour of bisecting code later. A measurement taken with +`src/locales/` dirty is suspect; one revert on the #1500 branch was made on +exactly such a measurement. Making the `.po` files read-only does not help: the +`.wuchale` output goes stale instead. +*Should have:* the compiled output regenerated whenever the `.po` files change +under it, or a refusal to serve when the two disagree. Until then: delete +`src/locales/.wuchale` together with any `.po` reset and restart Vite (rule in +`AGENTS.md`). + +**Vite serves a stale pre-bundle after a dependency's `dist` is edited.** +Editing an installed package's `dist/index.js` to instrument it is invisible +until the Vite cache directory is cleared; one measurement showed zero events +for that reason alone. +*Should have:* said which pre-bundle it was serving and when it was built. + +**A dependency install borrowed from another checkout.** +Every `node_modules` under `browser/` was a symlink into a different branch's +checkout, so this branch's `pnpm-lock.yaml` had never been installed: +`browser/patches/loro-prosemirror@0.4.3.patch` was not applied (the inline +editors dropped characters) and the production build died with +`null pointer passed to rust`, failing every `@smoke` test. Both read as product +bugs. A real `pnpm install --frozen-lockfile` fixed both. +*Should have:* the runner refusing to start when `node_modules` does not match +the lockfile, as `test-server` already refuses a stale binary. + +**`normalizeSubject('')` resolved to the server root.** +`useResource(x ?? '')` and every render before the drive setting hydrated +fetched `http://host/`, which 404s on a DID-drive server and logged two console +errors each time. It failed the zero-diagnostics gate in four specs and looked +like an architectural "nothing lives at the root" problem. +*Fixed 2026-09-16 (027c090) — the empty string is not a subject.* + **No `webServer` block, so every spec fails on a dead port.** With no dev server on 6747 the whole suite fails at `page.goto` with `ERR_CONNECTION_REFUSED`, which reads as a suite-wide code regression. diff --git a/planning/start-server-9897.sh b/planning/start-server-9897.sh deleted file mode 100755 index bd4bda8ea4..0000000000 --- a/planning/start-server-9897.sh +++ /dev/null @@ -1,36 +0,0 @@ -#!/bin/bash -# Start the e2e atomic-server on 9897 and do not return until it answers HTTP 200. -# redb's lock outlives the listening socket, so retry the spawn, not just the wait. -set -u -DATA=/private/tmp/atomic-hosting-node -BIN=/private/tmp/atomic-website-rust-build/debug/atomic-server -LOG=/tmp/server-9897.log - -if curl -s --max-time 2 -o /dev/null -w '' http://localhost:9897/ 2>/dev/null; then - code=$(curl -s --max-time 2 -o /dev/null -w "%{http_code}" http://localhost:9897/) - [ "$code" = "200" ] && { echo "already up"; exit 0; } -fi - -for attempt in 1 2 3 4 5 6; do - cd "$DATA" || exit 1 - ATOMIC_WEBSITE_ORIGIN=http://sites.localhost:9897 \ - ATOMIC_INITIALIZE=true \ - ATOMIC_INTEGRATION_PROXY_URL=http://127.0.0.1:19090 \ - TENANT_SECRET='bW9jay10ZW5hbnQ.mock-signature' \ - ATOMIC_INTEGRATION_FRONTEND_ORIGIN=http://localhost:6763 \ - nohup "$BIN" --port 9897 --ip 127.0.0.1 \ - --data-dir "$DATA/data" --config-dir "$DATA/config" --cache-dir "$DATA/cache" \ - > "$LOG" 2>&1 & - for i in $(seq 1 25); do - code=$(curl -s --max-time 2 -o /dev/null -w "%{http_code}" http://localhost:9897/ 2>/dev/null) - [ "$code" = "200" ] && { echo "server up (attempt $attempt, ${i}s)"; exit 0; } - sleep 1 - done - if grep -q "Database already open" "$LOG"; then - echo "attempt $attempt: redb lock still held, retrying" - sleep 5 - continue - fi - echo "attempt $attempt: did not become healthy"; tail -5 "$LOG"; sleep 3 -done -echo "FAILED to start server"; tail -20 "$LOG"; exit 1 diff --git a/planning/unified-data-layer.md b/planning/unified-data-layer.md index 9986cb9bf9..937ff38e6b 100644 --- a/planning/unified-data-layer.md +++ b/planning/unified-data-layer.md @@ -73,9 +73,14 @@ See [unify-resource-dirty-signals.md](./unify-resource-dirty-signals.md). regression before each flow change. Keep read and persistence state separate. - [ ] Audit direct `addResource` producers before further ingress consolidation. Preserve normalization, alias handling, echo imports/dedup, unsaved local state, - atomic persistence and notification ordering. + atomic persistence and notification ordering. Counted 2026-09-18: no non-test + `addResource(` call site remains in `browser/data-browser/src` (one in a + `.test.ts`); the remaining producers are inside `browser/lib/src`. - [ ] Review remaining Resource signing/retry responsibilities and `CommitBuilder` - consumers as a separate change. Preserve immutable signed envelopes and export + consumers as a separate change. Counted 2026-09-18: 31 `CommitBuilder` + references outside `*.test.ts`, all in `browser/lib/src` (`commit.ts` 18, + `resource.ts` 10, `store.ts` 2, `test-store.ts` 1), none in `browser/react/src` + or `browser/data-browser/src`. Preserve immutable signed envelopes and export baselines; do not conflate queued writes with persisted local durability. - [ ] Evaluate whether a persistence facade removes meaningful duplication. Existing atomic writes must remain intact; index-only seeding needs an explicit path. diff --git a/planning/website-publishing.md b/planning/website-publishing.md index db20a295b9..af545d86b8 100644 --- a/planning/website-publishing.md +++ b/planning/website-publishing.md @@ -1,9 +1,9 @@ # Website publishing: self-hosted first, managed hosting alongside it -Status: first FOSS implementation, built on the Assistant website prototype. - -For current implementation status, verification limits and prioritized follow-up -work, read [the implementation handoff](WEBSITE_HANDOFF.md). +Status: FOSS implementation on `develop` since #1500 (merged 2026-09-17), built +on the Assistant website prototype (#1498, still an open draft). The managed SaaS +adapter is not started. The open follow-ups from the #1500 handoff are in +[Remaining work](#remaining-work) below. ## Contract and boundaries @@ -79,10 +79,12 @@ Create a website and click **Publish site**. Later edits go live with **Update site**. That single explicit action saves the draft snapshot, uploads it and conditionally activates it. Conflicting publication changes are refused; the current site stays available if saving/uploading fails. One blue primary button -is visible at a time. **Publishing options** contains version restore and -unpublish. The same **Publishing options** menu contains export controls; there is no separate versions panel. Status loads on -mount, after actions and when the browser window regains focus; no manual -refresh control is exposed. Action errors use `store.notifyError` for the +is visible at a time. Version restore, unpublish, export, Design with AI and +View site are entries in the standard resource More menu (registered through +`useCustomContextItems` in `WebsitePage.tsx` / `WebsiteHosting.tsx`); versions +open a shared dialog and there is no separate "Publishing options" menu or +versions panel. Status loads on mount, after actions and when the browser +window regains focus; no manual refresh control is exposed. Action errors use `store.notifyError` for the standard toast and logging pipeline. API routes (all control requests signed with the existing Atomic request proof): @@ -147,3 +149,66 @@ structured rich text and its normal autosave path. Source permissions and privat authoring checks apply before enabling either editor. The iframe has no scripts or credentials; React and writes are owned by the Atomic host. Changes update the draft; publication still requires Update site. + +## Remaining work + +Carried from the #1500 handoff (2026-09-15/17) when it was deleted. In +suggested order; none of these is a regression against `develop`. + +1. **Inline rich text in the preview iframe is not fully verified.** Focused + Chromium and Firefox runs pass, but not every control is covered: mention + selection and arrow navigation, bubble menus, link and colour popovers, image + upload, tables, drag handles, undo/redo, cross-block selection, long + documents. Some shared RTE components still reference the global + `document`/`window` or portal defaults; audit them against iframe ownership + rather than loosening the sandbox. Two known flakes: + - `website-inline-content` / `website-inline-rte`: characters intermittently + vanish or land in a new paragraph when typing at human speed. The main + document editor also commits about once per keystroke at that speed, so + the save scheduler is a suspect; `loro-prosemirror` logs "Cannot find the + loroNode" around the failures. + - `website-inline-rte`: a popup (mention list, link field) open inside the + iframe is destroyed when the server echoes the author's own commit. The + echo stamps `lastCommit` on the resource's `properties` map; `LoroSyncPlugin` + subscribes to the whole doc, so it rebuilds the ProseMirror document for a + change that never touched `doc`. **Do not re-attempt filtering non-local + events in `updateNodeOnLoroEvent`**: measured, it broke the mention list + (the Suggestion plugin relies on those rebuild transactions) and did not + stop the character drop. The tractable fix is to make the tiptap + Suggestion plugin re-establish itself after a rebuild; then the filter + becomes viable. Product decision pending. +2. **Preview speed, invalidation and actionable failures.** Measure the + "Preparing preview" path (resource fetch, image decode/optimisation, render, + blob URL setup) before optimising; avoid re-fetching every asset on unrelated + changes; preserve cursor/selection during background updates. Audit the + Up to date / Unpublished changes state against actual published output, + never infer "clean" from an unavailable source. Missing-resource errors + should offer repair or removal of the binding, not a dead end with a DID. +3. **Managed atomic-saas publishing** (the major missing deliverable): a + destination adapter with the same status/upload/preview/activate semantics, + account-to-drive ownership, hosting entitlement, durable compare-and-swap + activation, private object storage and a customer-content origin + independent of the managed drive process. Prove staging publication, + private preview, unauthorised denial, activation conflict, failed-build + preservation, rollback, unpublish, source drive offline and image delivery. + Live S3/managed delivery has not been demonstrated. +4. **Storage and delivery production work**: retention and reference-aware + GC (failed uploads, concurrent publication), quota enforcement and + cap-reached UX, backup and restore covering metadata and blobs, cache and + serving cost measurements, portable unpublished exports across devices, + documented manifest compatibility and limits. +5. **Reassess renderer/plugin abstractions** against arbitrary AI designs and + interactive apps; the static CSP blocks scripts, network, workers and + forms, and changing that needs an explicit runtime/security design. Forms + should reuse the implementation from PR #1281 once a compatible revision + is verified (its merge status was not rechecked). Site deployment, form + availability and private response rows stay independent; rollback must + not reopen forms or delete responses. +6. **Authoring polish**: document File embeds, folder-to-gallery, labels and + icons, responsive layout, accessibility. Screenshot capture and Assistant + drag/drop were discussed but are not established as delivered. + +Harness notes that only matter for these specs: the website preview iframe +wants an origin separate from the app, so the production (embedded-in-server) +topology CI uses breaks the website specs while the Vite topology passes them; +do not compare results across topologies mid-investigation.