Skip to content

Electron shell: host the browser panel as a real view, not a video stream - #47

Merged
duolahypercho merged 3 commits into
mainfrom
electron-shell
Aug 16, 2026
Merged

Electron shell: host the browser panel as a real view, not a video stream#47
duolahypercho merged 3 commits into
mainfrom
electron-shell

Conversation

@duolahypercho

@duolahypercho duolahypercho commented Aug 15, 2026

Copy link
Copy Markdown
Owner

Why

The BROWSER tab works, but it never stopped feeling remote — because it is.

Tauri hands us the OS webview (WKWebView on macOS, WebView2 on Windows), which cannot host a second browser. So the panel is a CDP screencast: Chrome paints, core re-encodes to JPEG, the client draws it to a canvas and maps clicks back through the letterbox. Every complaint about it traced to that pipeline existing rather than to a tunable inside it:

Symptom What it actually was
Blurry Encode + rescale. Raising cast width/quality fixed it twice; it came back both times.
"I can't click, I can't type" Coordinate mapping through a stale viewport — a bug class that only exists because coordinates need mapping.
Tab toggle looked like a refresh The view being torn down and rebuilt.
Not smooth Frame pacing on a re-encoded stream.

Electron bundles Chromium, so the panel can be a WebContentsView the window composites directly. Real pixels, real input, real scrolling, no encode step.

What this does not do

It adds the Electron shell alongside Tauri. src-tauri/ and pnpm desktop:build are untouched and still the shipping path. Retiring either is a later decision, and the streaming pipeline stays until then because the Tauri build is its only consumer.

What's here

  • electron/main.ts — spawns or attaches to core, opens the window on the port core serves, and hands the panel's CDP target id to browser_attach, so the agent's browser_* tools drive the same view the user sees. One browser with two drivers, not one each.
  • electron/browserPanel.ts — creates the view on a persistent partition, hides rather than destroys it (a destroyed view loses the page — that was the "refresh on tab toggle"), and captures downloads to ~/.cali/downloads.
  • electron/preload.ts — a narrow window.cali over contextBridge under sandbox: true. No generic invoke; one named method per operation. Bundled with esbuild because a sandboxed preload cannot require a sibling module and the failure is silent (window.cali === undefined at render time, no error anywhere).
  • BrowserTab.tsx — detects the bridge and, under Electron, skips the cast entirely. It becomes a geometry reporter that tells the shell where to put the view, and hides the view while a popover or dialog is open so native pixels never paint over the editor's own overlays.

How the UI was checked

scripts/compare-shells.mjs drives both shells over CDP and diffs the editor region — the concern here was a shell swap quietly wrecking the look, so it's measured rather than asserted.

The two agree inside the antialiasing floor, measured at 2,666–3,023 px; the threshold sits at 6,000. Tokens, the no-focus-ring rule, and the drag regions all render identically.

node scripts/compare-shells.mjs   # from client/

Verification

  • npx tsc -b --noEmit — clean
  • tsc -p tsconfig.electron.json --noEmit — clean
  • pnpm test904 passed / 74 files, green on two consecutive runs (one earlier run had a single timing flake in an unrelated retry test, which did not reproduce)
  • pnpm desktop:electron:build — packages unsigned to release-electron/

Scope

Two commits: the client shell, then the core half that makes it mean anything.

The core commit is scoped to the browser subsystem — rpc.rs and tools.rs take
only their browser hunks, so another session's in-flight computer.rs,
guardian.rs and spawn_ledger.rs work stays out. The one casualty: the
spawn_ledger registration they added to browser.rs is omitted, because that
module is not on main yet. It should be restored when their branch lands.

Not done

  • Signing and notarisation — entitlements are written, the identity is not.
  • No LLM turn has yet been observed choosing these tools against the native panel.

Plan and status: docs/plans/electron-shell.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DAV2UBU1PQ8mJFVWMhKyHx

…ream

The BROWSER tab works, but it never stopped feeling remote — because it is.
Tauri hands us the OS webview (WKWebView on macOS, WebView2 on Windows), which
cannot host a second browser, so the panel is a CDP screencast: Chrome paints,
core re-encodes to JPEG, the client draws it to a canvas and maps clicks back
through the letterbox. Every complaint traced to that pipeline rather than to a
tunable in it. Raising the cast width and quality fixed the blur twice and it
came back both times; the click and typing regressions were coordinate-mapping
bugs that only exist because coordinates need mapping at all.

Electron bundles Chromium, so the panel can be a `WebContentsView` the window
composites directly: real pixels, real input, real scrolling, no encode step.

This adds that shell **alongside** Tauri rather than replacing it. `src-tauri/`
and `pnpm desktop:build` are untouched and still the shipping path; retiring
either is a later decision, and the streaming pipeline stays until then because
the Tauri build is its only consumer.

What is here:

- `electron/main.ts` spawns or attaches to core, opens the window on the port
  core serves, and hands the panel's CDP target id to `browser_attach` so the
  agent's `browser_*` tools drive the same view the user sees. One browser, two
  drivers — not one each.
- `electron/browserPanel.ts` creates the view on a persistent partition, hides
  rather than destroys it (a destroyed view loses the page, which is what made
  tab toggles look like a refresh), and captures downloads to `~/.cali/downloads`.
- `electron/preload.ts` exposes a narrow `window.cali` over `contextBridge`
  under `sandbox: true` — no generic invoke, one named method per operation.
  It is bundled with esbuild because a sandboxed preload cannot `require` a
  sibling module, and the failure is silent.
- `BrowserTab.tsx` detects the bridge and, under Electron, skips the cast
  entirely: it becomes a geometry reporter that tells the shell where to put the
  view, and hides it while a popover or dialog is open so native pixels never
  paint over the editor's own overlays.

`scripts/compare-shells.mjs` drives both shells over CDP and diffs the editor
region, which is how the UI was checked rather than asserted: the two agree
inside the antialiasing floor measured at 2.6-3.0k pixels, so the threshold sits
at 6k. Nothing in the design system changed — the tokens, the no-focus-ring
rule and the drag regions all render identically, which was the thing most at
risk in a shell swap.

Scoped to the client on purpose. The core-side half (`browser_attach`,
`browser_downloads`, the attach-aware `close`) is held back because those hunks
live in `rpc.rs`, `tools.rs` and `agent.rs`, which are currently ~95% another
session's in-flight work; shipping them here would drag three unfinished modules
into this diff. Until that lands, the Electron shell runs against a core without
`browser_attach` and the panel simply does not receive agent input — the editor,
the viewport and every other tab are fully functional.

Not done: signing and notarisation (entitlements are written, the identity is
not), and no LLM turn has yet been observed choosing these tools against the
native panel.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DAV2UBU1PQ8mJFVWMhKyHx
@duolahypercho

Copy link
Copy Markdown
Owner Author

The red E2E check is pre-existing on main, not from this branch.

e2e/workflow.spec.ts:312 › routes concurrent editor calls to the owning page and session times out at 60s on run 31866375731 — that is ad8467e, the exact commit this branch is based on, with no changes of mine on it. Same test, same timeout, and the last three main runs are red the same way.

This branch touches no attachment-routing code; the client diff is 174 lines across BrowserTab, desktop.ts, workspace.ts and index.css. Rust core, Client and GitGuardian all pass here.

Worth fixing, but it belongs in its own change rather than being folded into this one.

…hrome

The client half of this branch made the BROWSER tab a real `WebContentsView`,
but core knew nothing about it. `Browsers::ensure` launches a Chrome on first
use, so the agent's `browser_*` tools drove an invisible browser of their own
while the user looked at a different one. Two browsers, and the agent's work
never appeared in the tab — which is worse than the screencast it replaced,
because at least the screencast was the same page.

`browser_attach` closes that gap. The shell reads its panel's devtools target
id and hands it to core, which adopts that target rather than starting anything.
One browser, two drivers.

The target id is passed, never discovered. Guessing by url or title would
eventually match the editor's own window, and core would start driving the app
instead of the page inside it.

Ownership is now explicit: `child` and `profile` are `Option`, `None` meaning
attached. `close` returns early in that case rather than sending `Browser.close`,
which would quit the application hosting the view — the editor the user is
working in. `live_attach_drives_a_browser_it_did_not_launch` asserts both halves
against a real Chrome, and reads the resulting url back over its *own* devtools
connection so it cannot pass on core merely believing it navigated. Removing the
ownership guard fails it on the second assertion, which is the one with teeth.

Also lands the three browser tools the client half already assumes:
`browser_mouse_move` (a delta, because a camera turns by motion rather than
destination), `browser_play` (keys held *while* looking — the other two are
strictly sequential, so strafing or aiming while advancing was impossible), and
`browser_downloads`. The registered-surface invariant moves 12 -> 15 and AGENTS.md
documents all three, which is what that assertion exists to force.

Scoped to the browser subsystem: `rpc.rs` and `tools.rs` get only their browser
hunks, and the `spawn_ledger` registration another session added to `browser.rs`
is left out because that module is not on main yet. It should be restored when
their branch lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DAV2UBU1PQ8mJFVWMhKyHx
@duolahypercho

Copy link
Copy Markdown
Owner Author

Follow-up: the core half is now in, because without it the branch was still running a second browser.

Caught in review: the client commit alone made the panel native, but core knew nothing about it. Browsers::ensure launches a Chrome on first use, so the agent's browser_* tools drove an invisible browser while the user looked at a different one — worse than the screencast it replaced, since at least the screencast was the same page. The shell was already POSTing browser_attach; core had no such method, and the handshake failed into a console.error nobody sees.

c157213 adds it. Ownership is now explicit — child and profile are Option, None meaning attached — and close returns early rather than sending Browser.close, which would quit the application hosting the view.

Proven rather than asserted: browser::tests::live_attach_drives_a_browser_it_did_not_launch runs against a real Chrome the test owns, and reads the resulting url back over its own devtools connection so it cannot pass on core merely believing it navigated. I mutation-checked the safety assertion — removing the ownership guard from close fails it with shutdown killed a browser core did not launch, so it has teeth.

cargo fmt --check, clippy --all-targets -D warnings, and the suite (887 + 2, 6 ignored) are green.

…on our line

Tauri's webview is a different engine per platform and none of them can host a
second browser, so the BROWSER tab could only ever be a video stream of a Chrome
running somewhere else. That was the reason for the Electron shell, and keeping
both meant every fix had to be made twice into a shell that could not have the
feature. So `client/src-tauri/` is gone, along with `@tauri-apps/*`, the `tauri`
script, and `compare-shells.mjs`, which needed two shells to compare.

Three things the removal had to carry rather than drop:

- **The folder picker.** `chooseNativeWorkspace` imported
  `@tauri-apps/plugin-dialog`. It goes through the shell's `chooseFolder` now,
  and `defaultPath` is threaded through the bridge because one caller passes the
  folder it expects and silently losing that is a regression nobody would notice
  until the panel opened in the wrong place.
- **The icons.** `icon.{icns,ico,png}` and the source SVG move to
  `build-electron/`, electron-builder's `buildResources`. Same artwork and the
  same `com.calicode.desktop` identifier, so this replaces the installed app
  rather than sitting beside it with a second `~/.cali`.
- **`scripts/desktop.sh`.** Rewritten, not deleted: it still owns
  `build|dev|install`, and it still builds with a *stable* local signing
  identity, because an ad-hoc signature is keyed to the binary hash and macOS
  drops the app's Desktop/Documents grants on every rebuild. Signing moved from a
  post-hoc `codesign --deep` to electron-builder's own — an Electron bundle's
  helpers and framework must be signed inside-out, and `--deep` does it in the
  wrong order.

**The traffic lights.** `trafficLightPosition` was copied from `tauri.conf.json`,
and the two shells do not measure it the same way: `y: 23` sat the lights ~10pt
below the sidebar's window-controls row, on a line of their own. Measured against
a real window, Electron places the group ~6.75pt above the buttons' visual
centre, so `y: 13` centres them at 20pt — the same line as the `h-10` row. On
screen the lights, the sidebar toggle, the history arrows and the tab strip now
share one centre to within 0.5pt, which is glyph shape rather than layout.

**The name.** `app.setName` gives the userData directory and `app.getName()`, and
the packaged bundle reports `CFBundleName`/`CFBundleDisplayName` of CaliCode with
the brand icon. It does *not* rename the menu bar in an unpackaged run — that
title comes from the running bundle's plist, which is Electron's own — so
`pnpm desktop:electron` still says "Electron". Documented rather than chased.

`SettingsPage.test.tsx` faked the desktop shell by defining `__TAURI_INTERNALS__`.
That global no longer exists, so the test was changed to install the Electron
bridge instead — a test edit, said out loud: it asserts the same thing about the
same titlebar, only through the detection that is actually left.

Verified by driving both builds and measuring the pixels: unpackaged and packaged
windows agree on the alignment, the packaged app carries the icon and the name,
and core drives the panel with no Chrome of its own — proven by pointing the
panel at a distinct url over CDP and watching `browser_status` report it back.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DAV2UBU1PQ8mJFVWMhKyHx
@duolahypercho

Copy link
Copy Markdown
Owner Author

Tauri is gone; Electron is the only shell (0ef10c6).

Keeping both meant every fix had to be made twice, into a shell that structurally could not have the browser panel. Removed: client/src-tauri/, @tauri-apps/*, the tauri script, and compare-shells.mjs (it needed two shells to compare). 80 files, +206/−5611.

Three things the removal carried rather than dropped:

  • Folder pickerchooseNativeWorkspace used @tauri-apps/plugin-dialog; it goes through the shell's chooseFolder now, with defaultPath threaded through the bridge so the one caller that passes a starting folder still gets it.
  • Icons — moved to build-electron/ (electron-builder's buildResources), same artwork and the same com.calicode.desktop identifier, so this replaces the installed app rather than sitting beside it.
  • scripts/desktop.sh — rewritten, not deleted. Still owns build|dev|install, still uses a stable local signing identity (an ad-hoc signature is keyed to the binary hash, so macOS drops the app's TCC grants every rebuild). Signing moved to electron-builder's own, because an Electron bundle must be signed inside-out and codesign --deep does it in the wrong order.

Traffic lights. trafficLightPosition had been copied from tauri.conf.json, and the shells don't measure it the same way — y: 23 sat the lights ~10pt below the sidebar's controls row, on their own line. Measured against a real window, Electron places the group ~6.75pt above the buttons' visual centre, so y: 13 centres them at 20pt. Lights, sidebar toggle, history arrows and tab strip now share one centre to within 0.5pt.

Name. Packaged bundle reports CFBundleName/CFBundleDisplayName = CaliCode with the brand icon. app.setName does not rename the menu bar unpackaged — that comes from the running bundle's plist — so pnpm desktop:electron still says "Electron". Documented rather than chased.

One test changed, said out loud: SettingsPage.test.tsx faked the shell with __TAURI_INTERNALS__. That global no longer exists, so it installs the Electron bridge instead — same assertion, same titlebar, through the detection that is left.

Verified by driving both builds and measuring pixels rather than asserting: unpackaged and packaged agree on alignment, the packaged app carries icon and name, and core drives the panel with no Chrome of its own (proven by pointing the panel at a distinct url over CDP and watching browser_status report it back).

Client and Rust core pass. E2E is still the pre-existing workflow.spec.ts:312 timeout that fails on main at this branch's base.

@duolahypercho
duolahypercho merged commit a562a94 into main Aug 16, 2026
3 of 4 checks passed
duolahypercho added a commit that referenced this pull request Aug 18, 2026
main retired Tauri (#47) while this branch was building starters, memory,
hooks and file-defined commands. Eleven files conflicted and the two
sides were newer in opposite places, so the resolution is not "take one
side".

main wins on the shell, because retiring Tauri is the deliberate newer
decision: `client/electron/*` (which both sides had added independently
— main's carries `app.setName`, the brand icon, and a
`trafficLightPosition` measured against a real window rather than
inherited from tauri.conf.json), `desktop.ts`, `workspace.ts` (the
`@tauri-apps/plugin-dialog` path is gone, `chooseFolder` takes
`defaultPath`), `package.json`, `index.css` (`data-tauri-drag-region`
becomes `data-drag-region`), and the plan doc.

This branch wins on the agent browser, because its `browser_play` is the
newer one: `recordFrames` and `act_while_held` capture frames *during*
the hold, and `nativeVirtualKeyCode` is deliberately absent — it was
being handed the Windows VK code, which on macOS is a different keycode
space, so every held key arrived as Numpad5 and auto-repeated ~20k times
in 600ms.

Kept from both: main's `browser_attach` and its `live_attach` test,
which is the Electron panel's whole contract, alongside this branch's
frame recording and keycode assertions. AGENTS.md took main's shell
narrative and the `browser_attach` bullet, minus the older duplicate
"Playing a game is a different verb" bullet it would have reintroduced
over this branch's newer one.

Verified on the merge: cargo fmt/clippy clean, 1075 Rust tests passed,
tsc clean, 941 client tests passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013CUFpThK67PNcJz16UvH8C
@duolahypercho
duolahypercho deleted the electron-shell branch August 18, 2026 02:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant