tutsmaker

An autonomous browser agent that learns a web app by using it, then writes the tutorial.

Give it a URL, a plain-English goal and (optionally) login credentials. It drives a real Chromium browser to accomplish the goal, screenshots every meaningful step, and compiles the session into an illustrated MDX tutorial.

curl -X POST http://localhost:3000/api/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://app.example.com",
    "prompt": "Create a tutorial on how to set up a store",
    "credentials": { "email": "user@example.com", "password": "hunter2" }
  }'

{ "slug": "a1b2c3d4", "status": "in progress", "tutorialUrl": "http://localhost:3000/api/tutorials/a1b2c3d4" }

The request returns immediately and the pipeline runs in the background. When it finishes, the tutorial URL serves a rendered HTML page with inline screenshots.

---

How it works

flowchart LR
  A[POST /api/generate] --> B[Launch Chromium]
  B --> C[Login agent<br/>optional]
  C --> D[Agent loop<br/>observe → decide → act]
  D --> E[raw-session.json]
  E --> F[MDX compiler]
  F --> G[tutorial.mdx + screenshots]

Loading

The agent loop is a straightforward observe/decide/act cycle, capped at maxSteps:

1. Observe — wait for the DOM to go quiet, then capture a screenshot and an ARIA snapshot of the page. Both are appended to the conversation as a single user message.
2. Decide — send the history to the LLM with the tool schema. The model returns tool calls.
3. Act — execute each tool against the live page, push the result back as a tool message.
4. Repeat until the model calls done or the step budget runs out.

Every step is recorded to raw-session.json, which is what the MDX compiler reads. Because the session is persisted separately from the tutorial, prose can be regenerated without re-driving the browser — see /recompile below.

Tools available to the agent

Tool	Purpose
click	Click by screenshot coordinates, or by ARIA role + name
fill	Type into a single field by ARIA role + name
fillForm	Fill several fields in one call — preferred for forms
select	Choose a dropdown option; handles native <select> and custom widgets
scroll	Scroll by a percentage of viewport height
goto	Navigate to a URL
keys	Press keys (Enter, Tab, Ctrl+A) or type into the focused element
wait	Pause for a fixed duration when the page is still loading
done	Signal completion with a summary

---

Notable engineering decisions

Most of the difficulty in a browser agent is not the LLM call — it is making a click land where the model thinks it will. These are the parts that took real work.

Credentials never reach the LLM. The system prompt is told only the names of available variables (<variable name="password" />). The model emits %password% tokens, and substituteVariables() swaps in the real value locally inside the tool executor, after the model has already responded. Secrets stay on the machine.

Clicks go through CDP, not Playwright. Playwright's actionability check walks the DOM tree, so it reports "element is not visible" for dropdown options rendered into a portal and positioned with CSS — a pattern used by most component libraries. Resolving the element's bounding box and dispatching Input.dispatchMouseEvent at its centre sends a real mouse event to that screen position, which is what a user actually does.

Device pixel ratio is locked to 1 in two places. On a Retina display the browser produces screenshots at 2× the viewport, so a coordinate the model reads off the image is double what CDP expects, and every click lands in the wrong place. deviceScaleFactor: 1 on the context handles most cases; an Emulation.setDeviceMetricsOverride CDP call covers headed Chrome, where the context setting alone is not always honoured. Screenshots are also taken with scale: 'css', and the PNG's real dimensions are parsed straight out of the IHDR chunk and logged so a mismatch is caught immediately rather than showing up as mysteriously misplaced clicks.

Element resolution falls back five ways. getByRole with an accessible name is correct but frequently insufficient in the wild, so resolveByRoleWithFallback() tries, in order: scoped inside an open dialog (so a same-named button behind a modal is never hit), accessible-name match, inner-text match, sole-element-of-that-role, and finally visible text — which catches custom components that render no role attribute at all.

Context stays flat as the session grows. Screenshots dominate token usage, so pruneImages() strips base64 data from every image except the most recent keepLastScreenshots, replacing it with a short text note. A 30-step session costs roughly what a 3-step session does on input tokens.

Waiting is based on DOM quiet, not timeouts. waitForDomStable() attaches a MutationObserver to the document; each mutation resets a debounce timer, and the page is considered ready once nothing has changed for stableMs. A hard timeout prevents apps with live-updating data or perpetual animation from blocking forever.

Providers are swappable behind one interface. LLMProvider exposes a single chat() method, and each provider owns its own message-format conversion, image-detail mapping and cost table. The agent loop and login handler contain no provider-specific code. Two quirks are handled inside the provider boundary: Gemini returns click coordinates normalised to 0–1000 and they are rescaled to viewport pixels, and Gemini's thinking models attach a thoughtSignature to function calls that must be round-tripped verbatim in history or the conversation breaks.

Cost is tracked per call. Both providers carry a per-million-token pricing table and report input tokens, output tokens and USD for every request, accumulated across the run and written into the session file.

---

Setup

Requires Node 18+.

npm install
npx playwright install chromium
cp .env.example .env    # add OPENAI_API_KEY and/or GEMINI_API_KEY
npm run dev

The server listens on port 3000 by default (PORT to override).

API

Method	Route	Description
POST	/api/generate	Start a run. Body: url, prompt, optional credentials. Returns a slug immediately.
GET	/api/tutorials/:slug	Rendered HTML tutorial with inline screenshots. 404 while still generating.
GET	/api/tutorials/:slug/recompile	Regenerate the prose from the saved session, without re-running the browser.

Configuration

Set in AGENT_CONFIG in src/index.ts:

Option	Default	Notes
model	gemini-3-flash-preview	Provider is inferred from the name unless llmProvider is set
mdxModel	gemini-3-flash-preview	Model used for the writing pass; can differ from the driving model
maxSteps	30	Step budget for the agent loop
headless	false	Watch it work, or run it headless
imageQuality	high	Screenshot detail hint passed to the vision model
keepLastScreenshots	2	How many recent screenshots keep their image data

Output

Each run writes to src/storage/{slug}/ (gitignored):

tutorial.mdx           the finished tutorial
images/step-NN.png     one screenshot per observed step
raw-session.json       full session log — every tool call, result, reasoning and cost

Project layout

src/
  index.ts             Express server and pipeline orchestration
  config.ts            config resolution and defaults
  agent/
    agentLoop.ts       observe → decide → act loop, session recording
    systemPrompt.ts    agent system prompt
  browser/
    launcher.ts        Chromium launch, viewport and DPR locking
    a11y.ts            ARIA snapshot capture with truncation
    domStable.ts       MutationObserver-based settle detection
    screenshot.ts      screenshot capture and PNG dimension checks
  llm/
    LLMProvider.ts     provider interface
    types.ts           provider-agnostic message and tool types
    pruneImages.ts     context-window management
    providers/         OpenAI and Gemini implementations
  login/
    loginHandler.ts    dedicated login agent, runs before the main loop
  mdx/
    mdxCompiler.ts     session log → MDX tutorial
  tools/
    index.ts           tool schemas and executor

Status

A working prototype, not a packaged library. Known gaps:

• No test suite.
• AnthropicProvider is declared in the provider factory but not implemented.
• Config lives in a constant in src/index.ts rather than being per-request.
• The login agent shares the main tool set, so it can technically wander; it is constrained by prompt and a 15-step cap rather than by a restricted schema.