Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -966,7 +966,7 @@ Text highlighting uses `web-highlighter` library. Code blocks use manual `<mark>

**Raw-HTML annotate:** the sandboxed viewer never mutates the visited page's DOM. Committed annotations render as numbered placed comment markers plus overlay-projected highlight rectangles inside a shadow-rooted fixed overlay host: the durable anchor data (element selector, text snapshot, normalized selected point) is persisted, and the markers/highlights are disposable projections re-resolved from it on every reconcile. Shift-click multi-select joins additional elements to one comment (`htmlAdditionalTargets`).

**Element context (agent-facing).** A pinpoint also captures a bounded description of the element at click time, `elementContext` on the annotation (`HtmlElementContext` in `packages/ui/types.ts`; extra targets carry a smaller one as `context`), built by `buildElementContext` in the bridge and re-validated by `parseHtmlElementContext` at the parent trust boundary (`useHtmlAnnotation.ts`). It is purely descriptive, never read by restore. Fields: tag, id, author classes, an ancestor `path`, explicit-or-implicit `role`, accessible `name`, an ALLOWLISTED attribute set (href/src scrubbed of query and fragment, `data:` truncated to its media type), rendered `text` (300), a collapsed HTML `outline` (600 chars; two child levels, then one, then a per-tag count, whichever first fits), child count, viewport `rect`, nearest `landmark` and `heading`, a `component` hint from `data-component`/`data-testid` ancestry (deliberately no React fiber reads), and in live-app sessions the `page` route and title. Never captured: form values, `on*` handlers, `style`, script/style/template contents, full innerHTML. Hard cap 2 KiB serialized per primary (1 KiB per extra target), shedding outline → text → attrs → classes → path → heading → landmark → component. The export (`elementContextExportBlock` in `packages/ui/utils/parser.ts`) prints a 4-backtick `html` fence of the outline plus `selector` / `path` / `role` · `name` · `component` / `attrs` / `text` / `box` / `near` lines under the comment; a text-less pinpoint's placeholder quote (`[element: Navigation]`) becomes `Feedback on the <nav> element — "Primary"` in the heading, while a pinpoint with real quoted text keeps its quote line and gains the block. Annotations without the field export byte-identically. The grouped live-app export omits the `route` line (the `## Page:` heading carries it); `exportAnnotationEntry` (one annotation, no number) includes it; it is a pure helper for hosts and tests, and the annotation panel's card chrome is unchanged. Share links drop `elementContext` exactly like anchors. The feedback archive records identity only (`elementTag`, `elementSelector`, `elementPath`, `elementRole`, `elementName`, `pageUrl`), never the outline. No `BRIDGE_PROTOCOL_VERSION` bump: the field is additive in both directions.
**Element context (agent-facing).** A pinpoint also captures a bounded description of the element at click time, `elementContext` on the annotation (`HtmlElementContext` in `packages/ui/types.ts`; extra targets carry a smaller one as `context`), built by `buildElementContext` in the bridge and re-validated at the parent trust boundary by `parseHtmlElementContext`, which lives in `packages/core/html-anchor.ts` beside `parseHtmlElementAnchor` (#1549); `useHtmlAnnotation.ts` imports and re-exports it rather than mirroring it, and the host persistence helpers in that same core module (`buildPersistedHtmlAnchor`, `projectHostThreads`) run it too, so a host that persists and projects through core keeps the field instead of dropping it on save. It is purely descriptive, never read by restore. Fields: tag, id, author classes, an ancestor `path`, explicit-or-implicit `role`, accessible `name`, an ALLOWLISTED attribute set (href/src scrubbed of query and fragment, `data:` truncated to its media type), rendered `text` (300), a collapsed HTML `outline` (600 chars; two child levels, then one, then a per-tag count, whichever first fits), child count, viewport `rect`, nearest `landmark` and `heading`, a `component` hint from `data-component`/`data-testid` ancestry (deliberately no React fiber reads), and in live-app sessions the `page` route and title. Never captured: form values, `on*` handlers, `style`, script/style/template contents, full innerHTML. Hard cap 2 KiB serialized per primary (1 KiB per extra target), shedding outline → text → attrs → classes → path → heading → landmark → component. Under the persisted anchor's own 16 KiB budget `buildPersistedHtmlAnchor` sheds contexts — per-target first, from the end, then the primary — BEFORE it drops any target, since a context is descriptive and re-derivable on the next click while a dropped target loses a marker the reviewer placed; `DEFAULT_HTML_ANCHOR_MAX_BYTES` is unchanged and the dropped-target counts still count targets only. The export (`elementContextExportBlock` in `packages/ui/utils/parser.ts`) prints a 4-backtick `html` fence of the outline plus `selector` / `path` / `role` · `name` · `component` / `attrs` / `text` / `box` / `near` lines under the comment; a text-less pinpoint's placeholder quote (`[element: Navigation]`) becomes `Feedback on the <nav> element — "Primary"` in the heading, while a pinpoint with real quoted text keeps its quote line and gains the block. Annotations without the field export byte-identically. The grouped live-app export omits the `route` line (the `## Page:` heading carries it); `exportAnnotationEntry` (one annotation, no number) includes it; it is a pure helper for hosts and tests, and the annotation panel's card chrome is unchanged. Both helpers also take `includeOutline` (default `true`): `false` prints the identity lines without the fenced outline, for model turns where the 600-char outline is the expensive part per annotation. Share links drop `elementContext` exactly like anchors. The feedback archive records identity only (`elementTag`, `elementSelector`, `elementPath`, `elementRole`, `elementName`, `pageUrl`), never the outline. No `BRIDGE_PROTOCOL_VERSION` bump: the field is additive in both directions.

**HTML and live-app interaction model:** raw-HTML sessions and live app sessions (`mode: "annotate-app"`) share one contract. Both open with pinpoint **armed** (`htmlAnnotateArmed` defaults to `true`, `packages/editor/App.tsx:493`; live sessions open armed like every other HTML surface, `App.tsx:2871`). `Esc` walks a ladder instead of exiting outright: a pending draft closes first, then the pinpoint hover outline clears, and only then does `Esc` drop the surface to **Interact**, where the bridge goes passive so clicks, forms, text selection, and SPA navigation reach the page natively (`packages/ui/components/html-viewer/bridge-script.ts:3066-3080`; committed markers stay visible and a marker click still opens its comment, and in Interact an open drag-comment draft still closes before `Esc` is handed back to the page). Vim owns its own ladder and is skipped here. The header **pen** button toggles Annotate/Interact (`packages/editor/components/AppHeader.tsx:386-404`, `aria-pressed`), as does `Mod+Shift+A` (`packages/ui/shortcuts/plan-review/htmlAnnotate.shortcuts.ts`) — a real toggle in BOTH directions, which is what makes it the answer to "Esc dropped me to Interact, how do I get back?". Disarming through either path tears down any pending draft, because the bridge's `set-annotate-mode(false)` handler clears every pending affordance (`bridge-script.ts:719-739`), exactly as the Esc ladder does. The bridge mirrors the chord inside the iframe on the capture phase and forwards it to the parent, so it works whichever document owns focus. Text drag-selection commenting is **always live**, on both surfaces and in both states, ungated from the armed flag and from the input method (`bridge-script.ts:384-386`, `:1420-1436`): while armed, a click pins an element and a drag selects text at the same time, and the one-shot `dragEndedClick` guard stops a completed drag's trailing click from re-pinning (`bridge-script.ts:1381-1390`).

Expand Down
218 changes: 218 additions & 0 deletions packages/core/html-anchor.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@ import { describe, expect, test } from "bun:test";
import {
buildPersistedHtmlAnchor,
DEFAULT_HTML_ANCHOR_MAX_BYTES,
MAX_ELEMENT_CONTEXT_BYTES,
MAX_HTML_ADDITIONAL_TARGETS,
parseHtmlElementContext,
projectHostThreads,
type HostThread,
} from "./html-anchor";
Expand Down Expand Up @@ -47,6 +49,98 @@ describe("buildPersistedHtmlAnchor", () => {
expect(Object.keys(reordered.anchor.htmlAdditionalTargets![0]!)).toEqual(["text", "label", "anchor"]);
});

test("rows without elementContext serialize byte-identically and pin wire fingerprint", () => {
// Failure caught: changing the serialization format or introducing undefined/extra keys
// for rows that do not carry elementContext, which breaks host wire fingerprints.
const withoutContext = {
originalText: "Text quote",
htmlAnchor: { selector: "p.lead", tagName: "p", text: "Text" },
htmlAdditionalTargets: [
{ text: "Target", label: "Label", anchor: { selector: "#t1", tagName: "div" } },
],
};
const result = buildPersistedHtmlAnchor(withoutContext);
expect(JSON.stringify(result.anchor)).toBe(JSON.stringify(withoutContext));
expect(Object.keys(result.anchor)).toEqual(["originalText", "htmlAnchor", "htmlAdditionalTargets"]);
expect("elementContext" in result.anchor).toBe(false);

// When elementContext IS present, it serializes strictly after htmlAdditionalTargets
const withContext = {
...withoutContext,
elementContext: { tag: "p", id: "lead-para" },
};
const resultWithContext = buildPersistedHtmlAnchor(withContext);
expect(Object.keys(resultWithContext.anchor)).toEqual([
"originalText",
"htmlAnchor",
"htmlAdditionalTargets",
"elementContext",
]);
});

test("shed order: per-target contexts shed first, then primary context, before dropping targets under small maxBytes", () => {
// Failure caught: dropping targets prematurely while expendable contexts remain,
// losing user-selected targets instead of shedding descriptive context.
const targetContext1 = { tag: "button", path: "body > form > button:nth-of-type(1)", outline: "<button>1</button>" };
const targetContext2 = { tag: "button", path: "body > form > button:nth-of-type(2)", outline: "<button>2</button>" };
const primaryContext = { tag: "form", path: "body > form#action-form", outline: "<form>…</form>" };

const source = {
originalText: "Save changes",
htmlAnchor: { selector: "form#action-form", tagName: "form" },
htmlAdditionalTargets: [
{ text: "Btn 1", anchor: { selector: "button.save", tagName: "button" }, context: targetContext1 },
{ text: "Btn 2", anchor: { selector: "button.cancel", tagName: "button" }, context: targetContext2 },
],
elementContext: primaryContext,
};

// Full anchor with all contexts
const full = buildPersistedHtmlAnchor(source);
const fullBytes = bytes(full.anchor);
expect(full.anchor.elementContext).toBeDefined();
expect(full.anchor.htmlAdditionalTargets?.[0]?.context).toBeDefined();
expect(full.anchor.htmlAdditionalTargets?.[1]?.context).toBeDefined();
expect(full.anchor.htmlAdditionalTargets?.length).toBe(2);

// Budget just small enough to force shedding the last target's context
const budget1 = fullBytes - 30;
const res1 = buildPersistedHtmlAnchor(source, { maxBytes: budget1 });
expect(bytes(res1.anchor)).toBeLessThanOrEqual(budget1);
expect(res1.anchor.htmlAdditionalTargets?.length).toBe(2);
expect(res1.anchor.htmlAdditionalTargets?.[0]?.context).toBeDefined();
// Last target's context was shed first
expect(res1.anchor.htmlAdditionalTargets?.[1]?.context).toBeUndefined();
expect(res1.anchor.elementContext).toBeDefined();
expect(res1.droppedTargets).toBe(0);

// Budget small enough to shed all per-target contexts and primary context, but keep targets
const baseWithoutContexts = buildPersistedHtmlAnchor({
originalText: source.originalText,
htmlAnchor: source.htmlAnchor,
htmlAdditionalTargets: [
{ text: "Btn 1", anchor: { selector: "button.save", tagName: "button" } },
{ text: "Btn 2", anchor: { selector: "button.cancel", tagName: "button" } },
],
});
const budgetNoContexts = bytes(baseWithoutContexts.anchor) + 10;
const res2 = buildPersistedHtmlAnchor(source, { maxBytes: budgetNoContexts });
expect(bytes(res2.anchor)).toBeLessThanOrEqual(budgetNoContexts);
// Both target contexts and primary context shed, but targets remain
expect(res2.anchor.htmlAdditionalTargets?.length).toBe(2);
expect(res2.anchor.htmlAdditionalTargets?.[0]?.context).toBeUndefined();
expect(res2.anchor.htmlAdditionalTargets?.[1]?.context).toBeUndefined();
expect(res2.anchor.elementContext).toBeUndefined();
expect(res2.droppedTargets).toBe(0);

// Even smaller budget: now targets are dropped from the end
const budgetDropTarget = bytes(baseWithoutContexts.anchor) - 20;
const res3 = buildPersistedHtmlAnchor(source, { maxBytes: budgetDropTarget });
expect(bytes(res3.anchor)).toBeLessThanOrEqual(budgetDropTarget);
expect(res3.anchor.htmlAdditionalTargets?.length).toBe(1);
expect(res3.sizeDroppedTargets).toBe(1);
});

test("a drag capture without an element anchor writes exactly the legacy shape", () => {
const result = buildPersistedHtmlAnchor({ originalText: "plain quote" });
expect(result.anchor).toEqual({ originalText: "plain quote" });
Expand Down Expand Up @@ -224,6 +318,56 @@ describe("projectHostThreads", () => {
expect(gamma?.htmlAdditionalTargets).toEqual([{ label: "Button", text: "Go", anchor: { selector: "#go", tagName: "button", text: "Go" } }]);
});

test("projection carries validated primary elementContext and target contexts", () => {
// Failure caught: host panels and copy actions dropping elementContext and per-target
// context during projection from stored host rows onto viewer annotations.
const threadWithContext: HostThread = {
id: "ctx1",
originalText: "Quoted",
htmlAnchor: { selector: "#main", tagName: "main" },
elementContext: {
tag: "MAIN",
id: "main-content",
classes: ["content"],
role: "main",
},
htmlAdditionalTargets: [
{
text: "Extra target",
label: "Section",
anchor: { selector: "section#s1", tagName: "section" },
context: { tag: "SECTION", id: "s1" },
},
],
};
const projected = projectHostThreads([threadWithContext]);
expect(projected.length).toBe(1);
expect(projected[0]?.elementContext).toEqual({
tag: "main",
id: "main-content",
classes: ["content"],
role: "main",
});
expect(projected[0]?.htmlAdditionalTargets?.[0]?.context).toEqual({
tag: "section",
id: "s1",
});

// Malformed context fails closed to undefined without dropping the annotation
const threadWithGarbage: HostThread = {
id: "ctx2",
originalText: "Quoted 2",
elementContext: { tag: "" } as any,
htmlAdditionalTargets: [
{ text: "Extra", context: null as any },
],
};
const projectedGarbage = projectHostThreads([threadWithGarbage]);
expect(projectedGarbage[0]?.elementContext).toBeUndefined();
expect(projectedGarbage[0]?.htmlAdditionalTargets?.[0]?.context).toBeUndefined();
expect(projectedGarbage[0]?.originalText).toBe("Quoted 2");
});

test("is pure: the same input projects the same output and never mutates it", () => {
const frozen = JSON.stringify(rows);
const a = projectHostThreads(rows, { openOnly: true });
Expand All @@ -232,3 +376,77 @@ describe("projectHostThreads", () => {
expect(JSON.stringify(rows)).toBe(frozen);
});
});

describe("parseHtmlElementContext", () => {
test("accepts what the bridge builds and fails closed on garbage", () => {
// Failure caught: forged or malformed bridge context messages crashing the host
// or injecting unvalidated page-controlled content into storage.
expect(parseHtmlElementContext(null)).toBeUndefined();
expect(parseHtmlElementContext(undefined)).toBeUndefined();
expect(parseHtmlElementContext("not-an-object")).toBeUndefined();
expect(parseHtmlElementContext({})).toBeUndefined();
expect(parseHtmlElementContext({ tag: "" })).toBeUndefined();
expect(parseHtmlElementContext({ tag: " " })).toBeUndefined();

// Valid context with lowercased tag and parsed fields
const valid = parseHtmlElementContext({
tag: "BUTTON",
id: "submit-btn",
classes: ["btn", "btn-primary"],
path: "body > form > button",
role: "button",
name: "Submit form",
attrs: [["type", "submit"], ["aria-label", "Submit form"], ["onclick", "steal()"]],
text: "Submit",
outline: "<button type=\"submit\">\n```\nSubmit\n</button>",
children: 1.9,
rect: { x: 10, y: 20, w: 100, h: 40, vw: 1280, vh: 800 },
landmark: "main",
heading: "h1 \"Checkout\"",
component: "data-component=SubmitBtn",
page: { url: "/checkout?step=2", title: "Checkout" },
});
expect(valid).toBeDefined();
expect(valid?.tag).toBe("button");
expect(valid?.id).toBe("submit-btn");
expect(valid?.classes).toEqual(["btn", "btn-primary"]);
expect(valid?.path).toBe("body > form > button");
expect(valid?.role).toBe("button");
expect(valid?.name).toBe("Submit form");
// Non-allowlisted attribute onclick was dropped
expect(valid?.attrs).toEqual([["type", "submit"], ["aria-label", "Submit form"]]);
expect(valid?.text).toBe("Submit");
// Fenced backticks defused
expect(valid?.outline).not.toContain("```");
expect(valid?.outline).toContain("'''");
// Children floored
expect(valid?.children).toBe(1);
expect(valid?.rect).toEqual({ x: 10, y: 20, w: 100, h: 40, vw: 1280, vh: 800 });
expect(valid?.landmark).toBe("main");
expect(valid?.heading).toBe("h1 \"Checkout\"");
expect(valid?.component).toBe("data-component=SubmitBtn");
expect(valid?.page).toEqual({ url: "/checkout?step=2", title: "Checkout" });

// Non-finite rect is dropped whole
const badRect = parseHtmlElementContext({ tag: "div", rect: { x: NaN, y: 0, w: 10, h: 10, vw: 100, vh: 100 } });
expect(badRect?.rect).toBeUndefined();

// Unknown keys dropped
const extraKeys = parseHtmlElementContext({ tag: "div", unknownKey: "evil", innerHTML: "<script>" });
expect(extraKeys).toEqual({ tag: "div" });
expect("unknownKey" in (extraKeys ?? {})).toBe(false);

// Bounds enforced: contextBytes <= MAX_ELEMENT_CONTEXT_BYTES
const huge = "x".repeat(5000);
const oversized = parseHtmlElementContext({
tag: "div",
text: huge,
outline: huge,
path: huge,
classes: Array(30).fill("c"),
attrs: Array(30).fill(["title", "t"]),
});
expect(oversized).toBeDefined();
expect(bytes(oversized)).toBeLessThanOrEqual(MAX_ELEMENT_CONTEXT_BYTES);
});
});
Loading