diff --git a/CHANGELOG.md b/CHANGELOG.md index 8f42d39..a5a80a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,21 @@ All notable changes to this project are documented here. the Markdown-to-sanitized-HTML renderer (previously private to `Form.svelte`) into `companion/src/lib/markdown.ts` so both widgets share one DOMPurify allowlist instead of drifting. +- **New form field `annotated_image` — mark a point or region on an + image (#24).** When the answer the agent needs is spatial ("where should + the logo go?", "which part do I crop?", "point at the bug in this + screenshot"), words are a poor carrier. The field shows an image and lets + the user mark it directly: `mode: "point"` (click a crosshair marker, + default), `mode: "region"` (drag a rectangle), or `mode: "both"` (a + Point/Region toggle; both can be set). The `src` reuses the existing image + resolution (absolute / `~/` local path inlined by the bridge, `http(s)://` + fetched Mac-side, or `data:`). The result — under the field `name` — + carries **normalized 0..1** coordinates: `{point: {x, y} | null, region: + {x, y, w, h} | null, natural: {width, height} | null}`, resolution + independent, with `natural` (the image's intrinsic pixel size) so the agent + can recover pixel coordinates losslessly. `required` gates the submit + action on an annotation being present. Documented in the skill field + catalog and mirrored in the Rust and Python `form` tool descriptions. ## [0.8.3] — 2026-07-29 diff --git a/companion/src-tauri/src/dialog.rs b/companion/src-tauri/src/dialog.rs index 87bba52..4ec54ea 100644 --- a/companion/src-tauri/src/dialog.rs +++ b/companion/src-tauri/src/dialog.rs @@ -242,6 +242,16 @@ pub fn estimate_dialog_size(spec: &serde_json::Value) -> (f64, f64) { (w, (rows * 130.0 + 30.0).max(160.0)) } "image" => (BASE_W, 220.0), + "annotated_image" => { + // Interactive: image + tool/clear toolbar + coordinate readout. + // Reserve extra vertical room over a plain `image` so the stage + // isn't cramped and the drag surface stays usable. + let img_h = field + .get("max_height") + .and_then(|v| v.as_f64()) + .unwrap_or(320.0); + (BASE_W, (img_h + 90.0).min(560.0)) + } "tree" => (BASE_W, 240.0), "list" => { let items = field diff --git a/companion/src-tauri/src/http.rs b/companion/src-tauri/src/http.rs index a4abcdc..e87827e 100644 --- a/companion/src-tauri/src/http.rs +++ b/companion/src-tauri/src/http.rs @@ -771,7 +771,8 @@ async fn update( const KNOWN_FIELD_KINDS: &[&str] = &[ "text", "password", "secret", "number", "select", "checkbox", "slider", "date", "datetime", "date_range", "color", "static_text", "markdown", - "image", "mermaid", "wireframe", "image_grid", "list", "table", "tree", + "image", "annotated_image", "mermaid", "wireframe", "image_grid", "list", + "table", "tree", ]; /// Validate a dialog spec *before* any window is created (v0.4.46, @@ -1267,6 +1268,16 @@ mod validate_tests { assert!(validate_spec(&spec).is_ok()); } + #[test] + fn accepts_form_with_annotated_image() { + // #24: the annotated_image field must pass spec validation so the + // frontend widget gets a chance to render it. + let spec = json!({"kind":"form","fields":[ + {"kind":"annotated_image","name":"spot","src":"~/shot.png","mode":"point"} + ]}); + assert!(validate_spec(&spec).is_ok()); + } + #[test] fn accepts_form_with_tabs() { let spec = json!({"kind":"form","tabs":[ diff --git a/companion/src-tauri/src/mcp.rs b/companion/src-tauri/src/mcp.rs index 9940d15..f550559 100644 --- a/companion/src-tauri/src/mcp.rs +++ b/companion/src-tauri/src/mcp.rs @@ -333,7 +333,7 @@ fn tools_list() -> Value { }, { "name": "form", - "description": "Whenever the user needs to provide ≥ 2 related inputs, or any single input that doesn't belong in chat (secret, date/datetime/range, bounded number, sortable ranking, multi-select, color pick, table-row triage with column context, image confirm/grid), call this tool instead of typing the questions one by one. Fields: text, password, secret, number, select, checkbox, slider, date, datetime, date_range, color, static_text, markdown, image, mermaid, wireframe, image_grid, list, table, tree. **File-write / secret capture (#135):** any input field may carry an optional `target` to write the entered value to a file ON THE HOST THE AGENT RUNS ON when the user submits (the affirmative button IS the per-write approval; the user sees the path first): `{\"kind\":\"secret\",\"name\":\"pat\",\"label\":\"GitHub PAT\",\"target\":{\"mode\":\"create\",\"path\":\"~/.github_tokens/byte5ai\",\"perm\":\"0600\",\"overwrite\":true}}`. `mode`: `create` (write raw value; needs `overwrite:true` to clobber) or `substitute` (replace a `placeholder` that occurs exactly once in an existing file — for YAML/TOML/INI/etc; choose a DISTINCTIVE sentinel that can't collide with real file content, e.g. `__AIUI_SECRET_GITHUB_PAT__`, not a common word — if it occurs 0 or >1 times the write is refused with an error, never misapplied to the wrong spot). A `secret`-kind field is **write-only**: its value is NEVER returned to you (result carries only `{written, target, bytes}`); use it precisely so a credential the user types never enters this conversation. Non-secret fields with a `target` are written AND returned. The destination is always the agent's own host: the aiui module already running there (the native app locally, the bridge on a remote SSH session) performs the write as a LOCAL file operation, so `create` and `substitute` both work identically local and remote — and you cannot target a foreign host. Errors come back as `{written:false, error}`. Group long forms with `tabs: [{label, fields: [...]}]` (one submit, all tabs validated). Footer actions are top-level on the form (`actions: [...]`), NOT inside a tab — they always render at the window's bottom. Action variants: primary (blue), success (green), destructive (red). Returns {cancelled, action?, values}. For yes/no, use `confirm`. For one-of-N pick, use `ask`. Sortable list field shape (most common stumble — always include `value` per item): {\"kind\":\"list\",\"name\":\"rank\",\"label\":\"Sortieren\",\"sortable\":true,\"items\":[{\"label\":\"A\",\"value\":\"a\"},{\"label\":\"B\",\"value\":\"b\"}]}. Image fields (`image`, `image_grid`, list-item `thumbnail`): `src` accepts (1) an absolute or `~/`-rooted local path — aiui's bridge on YOUR host reads it and inlines as `data:`; (2) an `http(s)://` URL — Mac-companion fetches and inlines; (3) a `data:` URL — pass through. Pick the path form when the file is on disk on your host. Relative paths and cross-host paths don't resolve. Never base64-roundtrip through a shell pipeline — build the `data:` URL in your runtime. For schematic visualisations (flowcharts, sequence/state diagrams, gantt, mind-maps) use the `mermaid` field instead of ASCII art: `{\"kind\":\"mermaid\",\"source\":\"graph TD; A --> B; B --> C\"}`. For UI-layout mockups (dashboard tiles, hardware-UI panels, login screens, anything with fixed-position boxes-and-labels) use the `wireframe` field — declarative panel grid, NOT ASCII boxes-and-pipes: `{\"kind\":\"wireframe\",\"columns\":3,\"panels\":[{\"title\":\"STATUS\",\"content\":\"Tiefe: 18 m\\nKurs: 270°\",\"col_span\":1},{\"title\":\"EMPFANG\",\"content\":\"14:32 [STARK]…\",\"col_span\":2}]}`. Each panel has optional `title` (uppercase header), `content` (multi-line monospace text, escape `\\n`), `col_span`/`row_span` (default 1), and `tone` (\"default\"/\"muted\"/\"highlight\"). See the aiui skill for the full field catalog. **This tool blocks until the user submits or cancels. Response can take minutes (longer for complex forms) — do not assume aiui is broken on slow response, the user is filling the form. The companion sends MCP progress notifications every ~10 s while waiting.**", + "description": "Whenever the user needs to provide ≥ 2 related inputs, or any single input that doesn't belong in chat (secret, date/datetime/range, bounded number, sortable ranking, multi-select, color pick, table-row triage with column context, image confirm/grid), call this tool instead of typing the questions one by one. Fields: text, password, secret, number, select, checkbox, slider, date, datetime, date_range, color, static_text, markdown, image, annotated_image, mermaid, wireframe, image_grid, list, table, tree. **File-write / secret capture (#135):** any input field may carry an optional `target` to write the entered value to a file ON THE HOST THE AGENT RUNS ON when the user submits (the affirmative button IS the per-write approval; the user sees the path first): `{\"kind\":\"secret\",\"name\":\"pat\",\"label\":\"GitHub PAT\",\"target\":{\"mode\":\"create\",\"path\":\"~/.github_tokens/byte5ai\",\"perm\":\"0600\",\"overwrite\":true}}`. `mode`: `create` (write raw value; needs `overwrite:true` to clobber) or `substitute` (replace a `placeholder` that occurs exactly once in an existing file — for YAML/TOML/INI/etc; choose a DISTINCTIVE sentinel that can't collide with real file content, e.g. `__AIUI_SECRET_GITHUB_PAT__`, not a common word — if it occurs 0 or >1 times the write is refused with an error, never misapplied to the wrong spot). A `secret`-kind field is **write-only**: its value is NEVER returned to you (result carries only `{written, target, bytes}`); use it precisely so a credential the user types never enters this conversation. Non-secret fields with a `target` are written AND returned. The destination is always the agent's own host: the aiui module already running there (the native app locally, the bridge on a remote SSH session) performs the write as a LOCAL file operation, so `create` and `substitute` both work identically local and remote — and you cannot target a foreign host. Errors come back as `{written:false, error}`. Group long forms with `tabs: [{label, fields: [...]}]` (one submit, all tabs validated). Footer actions are top-level on the form (`actions: [...]`), NOT inside a tab — they always render at the window's bottom. Action variants: primary (blue), success (green), destructive (red). Returns {cancelled, action?, values}. For yes/no, use `confirm`. For one-of-N pick, use `ask`. Sortable list field shape (most common stumble — always include `value` per item): {\"kind\":\"list\",\"name\":\"rank\",\"label\":\"Sortieren\",\"sortable\":true,\"items\":[{\"label\":\"A\",\"value\":\"a\"},{\"label\":\"B\",\"value\":\"b\"}]}. Image fields (`image`, `image_grid`, list-item `thumbnail`): `src` accepts (1) an absolute or `~/`-rooted local path — aiui's bridge on YOUR host reads it and inlines as `data:`; (2) an `http(s)://` URL — Mac-companion fetches and inlines; (3) a `data:` URL — pass through. Pick the path form when the file is on disk on your host. Relative paths and cross-host paths don't resolve. Never base64-roundtrip through a shell pipeline — build the `data:` URL in your runtime. To have the user MARK a spot on an image (logo placement, crop hint, bug location) use `annotated_image`: `{\"kind\":\"annotated_image\",\"name\":\"spot\",\"src\":\"~/shot.png\",\"mode\":\"point\"}` — `mode` is `point` (click one marker, default), `region` (drag a rectangle), or `both` (user flips a Point/Region tool). `src` follows the same resolution rules as `image`. Returns normalized 0..1 coords under the field name: `{\"point\":{\"x\",\"y\"}|null,\"region\":{\"x\",\"y\",\"w\",\"h\"}|null,\"natural\":{\"width\",\"height\"}|null}` — multiply by `natural` for pixels. For schematic visualisations (flowcharts, sequence/state diagrams, gantt, mind-maps) use the `mermaid` field instead of ASCII art: `{\"kind\":\"mermaid\",\"source\":\"graph TD; A --> B; B --> C\"}`. For UI-layout mockups (dashboard tiles, hardware-UI panels, login screens, anything with fixed-position boxes-and-labels) use the `wireframe` field — declarative panel grid, NOT ASCII boxes-and-pipes: `{\"kind\":\"wireframe\",\"columns\":3,\"panels\":[{\"title\":\"STATUS\",\"content\":\"Tiefe: 18 m\\nKurs: 270°\",\"col_span\":1},{\"title\":\"EMPFANG\",\"content\":\"14:32 [STARK]…\",\"col_span\":2}]}`. Each panel has optional `title` (uppercase header), `content` (multi-line monospace text, escape `\\n`), `col_span`/`row_span` (default 1), and `tone` (\"default\"/\"muted\"/\"highlight\"). See the aiui skill for the full field catalog. **This tool blocks until the user submits or cancels. Response can take minutes (longer for complex forms) — do not assume aiui is broken on slow response, the user is filling the form. The companion sends MCP progress notifications every ~10 s while waiting.**", "inputSchema": { "type": "object", "required": ["title"], diff --git a/companion/src/lib/widgets/Form.svelte b/companion/src/lib/widgets/Form.svelte index 9d35fae..341efcf 100644 --- a/companion/src/lib/widgets/Form.svelte +++ b/companion/src/lib/widgets/Form.svelte @@ -65,6 +65,21 @@ | { kind: "static_text"; text: string; tone?: "info" | "warn" | "muted" } | { kind: "markdown"; text: string } | { kind: "image"; src: string; label?: string; alt?: string; max_height?: number } + | { + kind: "annotated_image"; + name: string; + src: string; + label?: string; + alt?: string; + /** point → single marker, region → rectangle, both → user picks a tool. Default "point". */ + mode?: "point" | "region" | "both"; + max_height?: number; + required?: boolean; + default?: { + point?: { x: number; y: number }; + region?: { x: number; y: number; w: number; h: number }; + }; + } | { kind: "mermaid"; source: string; label?: string; max_height?: number } | { kind: "wireframe"; @@ -217,6 +232,14 @@ }; case "image_grid": return { selected: [...(f.default_selected ?? [])] }; + case "annotated_image": + // Normalized (0..1) coordinates. `natural` is filled in once the + // image loads so the agent can recover pixel coordinates losslessly. + return { + point: f.default?.point ?? null, + region: f.default?.region ?? null, + natural: null as { width: number; height: number } | null, + }; case "tree": return { selected: [...(f.default_selected ?? [])], @@ -319,6 +342,128 @@ values[name] = { ...g, selected }; } + // --- annotated image ---------------------------------------------------- + // A single point marker and/or a rectangular region, both stored as + // normalized 0..1 coordinates relative to the *displayed* image (which, + // because the overlay exactly covers the , are also fractions of the + // natural image — resolution-independent). `mode` decides which gestures + // are available; in "both" the user flips an explicit Point/Region tool so + // the gesture is never ambiguous. + type AnnValue = { + point: { x: number; y: number } | null; + region: { x: number; y: number; w: number; h: number } | null; + natural: { width: number; height: number } | null; + }; + type AnnField = Extract; + // Below this drag distance (in normalized units) a region gesture counts as + // a stray click and is discarded rather than committing a degenerate rect. + const ANN_MIN_REGION = 0.01; + + let annTool = $state>({}); + let annDrag = $state<{ + name: string; + tool: "point" | "region"; + startX: number; + startY: number; + prevRegion: AnnValue["region"]; + } | null>(null); + + function annActiveTool(f: AnnField): "point" | "region" { + if (f.mode === "region") return "region"; + if (f.mode !== "both") return "point"; + return annTool[f.name] ?? "point"; + } + + function annSetTool(f: AnnField, tool: "point" | "region") { + annTool = { ...annTool, [f.name]: tool }; + } + + function annNormFromEvent(e: PointerEvent, stage: HTMLElement): { x: number; y: number } { + const rect = stage.getBoundingClientRect(); + if (rect.width <= 0 || rect.height <= 0) return { x: 0, y: 0 }; + const x = Math.min(1, Math.max(0, (e.clientX - rect.left) / rect.width)); + const y = Math.min(1, Math.max(0, (e.clientY - rect.top) / rect.height)); + return { x: round4(x), y: round4(y) }; + } + + function round4(n: number): number { + return Math.round(n * 1e4) / 1e4; + } + + function annOnImageLoad(name: string, img: HTMLImageElement) { + if (!img.naturalWidth || !img.naturalHeight) return; + const v = values[name] as AnnValue; + values[name] = { ...v, natural: { width: img.naturalWidth, height: img.naturalHeight } }; + } + + function annPointerDown(f: AnnField, e: PointerEvent, stage: HTMLElement) { + // Only react to the primary (left / touch / pen) button. + if (e.button !== 0) return; + e.preventDefault(); + stage.setPointerCapture?.(e.pointerId); + const tool = annActiveTool(f); + const p = annNormFromEvent(e, stage); + const v = values[f.name] as AnnValue; + annDrag = { name: f.name, tool, startX: p.x, startY: p.y, prevRegion: v.region }; + if (tool === "point") { + values[f.name] = { ...v, point: p }; + } else { + // Start a zero-size region; it grows on move. + values[f.name] = { ...v, region: { x: p.x, y: p.y, w: 0, h: 0 } }; + } + } + + function annPointerMove(f: AnnField, e: PointerEvent, stage: HTMLElement) { + if (!annDrag || annDrag.name !== f.name) return; + e.preventDefault(); + const p = annNormFromEvent(e, stage); + const v = values[f.name] as AnnValue; + if (annDrag.tool === "point") { + values[f.name] = { ...v, point: p }; + } else { + const x = Math.min(annDrag.startX, p.x); + const y = Math.min(annDrag.startY, p.y); + const w = Math.abs(p.x - annDrag.startX); + const h = Math.abs(p.y - annDrag.startY); + values[f.name] = { + ...v, + region: { x: round4(x), y: round4(y), w: round4(w), h: round4(h) }, + }; + } + } + + function annPointerUp(f: AnnField, e: PointerEvent, stage: HTMLElement) { + if (!annDrag || annDrag.name !== f.name) return; + e.preventDefault(); + stage.releasePointerCapture?.(e.pointerId); + const drag = annDrag; + annDrag = null; + if (drag.tool === "region") { + const v = values[f.name] as AnnValue; + const r = v.region; + // Discard a degenerate drag (a stray click) — restore what was there. + if (!r || r.w < ANN_MIN_REGION || r.h < ANN_MIN_REGION) { + values[f.name] = { ...v, region: drag.prevRegion }; + } + } + } + + function annClear(f: AnnField) { + const v = values[f.name] as AnnValue; + values[f.name] = { ...v, point: null, region: null }; + } + + function annHasAnnotation(f: AnnField): boolean { + const v = values[f.name] as AnnValue | undefined; + if (!v) return false; + const tool = annActiveTool(f); + if (f.mode === "point") return !!v.point; + if (f.mode === "region") return !!v.region; + // "both": either satisfies. Use tool only as a hint; presence wins. + void tool; + return !!v.point || !!v.region; + } + function toggleTreeExpand(name: string, value: string) { const t = values[name] as { selected: string[]; expanded: Set }; const expanded = new Set(t.expanded); @@ -361,6 +506,10 @@ const v = values[f.name] as { selected: string[] }; return v.selected.length > 0; } + if (f.kind === "annotated_image") { + if (!f.required) return true; + return annHasAnnotation(f); + } if (!("required" in f) || !f.required) return true; const v = values[(f as any).name]; return v !== undefined && v !== null && String(v).length > 0; @@ -463,6 +612,87 @@ {f.alt {#if f.label}
{f.label}
{/if} + {:else if f.kind === "annotated_image"} + {@const ann = values[f.name] as AnnValue} + {@const annMode = f.mode ?? "point"} +
+ {#if f.label}{/if} +
+ {#if annMode === "both"} +
+ + +
+ {:else} + + {annMode === "region" ? "Drag to mark a region" : "Click to mark a point"} + + {/if} + +
+ +
annPointerDown(f, e, e.currentTarget as HTMLElement)} + onpointermove={(e) => annPointerMove(f, e, e.currentTarget as HTMLElement)} + onpointerup={(e) => annPointerUp(f, e, e.currentTarget as HTMLElement)} + onpointercancel={(e) => annPointerUp(f, e, e.currentTarget as HTMLElement)} + > + {f.alt annOnImageLoad(f.name, e.currentTarget as HTMLImageElement)} + /> + +
+
+ {#if ann.point} + point {ann.point.x.toFixed(3)}, {ann.point.y.toFixed(3)} + {/if} + {#if ann.region} + region {ann.region.x.toFixed(3)}, {ann.region.y.toFixed(3)} · {ann.region.w.toFixed(3)}×{ann.region.h.toFixed(3)} + {/if} + {#if !ann.point && !ann.region} + No annotation yet + {/if} +
+
{:else if f.kind === "mermaid"} {:else if f.kind === "wireframe"} @@ -807,6 +1037,94 @@ text-align: center; } + /* --- annotated image --- */ + .annimg { display: flex; flex-direction: column; gap: 6px; } + .annimg-toolbar { + display: flex; + align-items: center; + gap: 8px; + flex-wrap: wrap; + } + .annimg-hint { font-size: 12px; color: var(--muted); } + .annimg-tools { display: inline-flex; gap: 0; border: 1px solid var(--border); border-radius: 7px; overflow: hidden; } + .annimg-tool { + background: var(--surface); + border: none; + border-radius: 0; + box-shadow: none; + padding: 4px 10px; + font-size: 12px; + color: var(--muted); + cursor: pointer; + } + .annimg-tool + .annimg-tool { border-left: 1px solid var(--border); } + .annimg-tool.active { background: var(--accent); color: var(--accent-fg); } + .annimg-clear { + margin-left: auto; + padding: 4px 10px; + font-size: 12px; + border: 1px solid var(--border); + border-radius: 7px; + background: var(--surface); + cursor: pointer; + } + .annimg-clear:disabled { opacity: 0.5; cursor: default; } + .annimg-stage { + position: relative; + width: fit-content; + max-width: 100%; + border-radius: 8px; + overflow: hidden; + border: 1px solid var(--border); + background: var(--surface); + cursor: crosshair; + touch-action: none; + user-select: none; + } + .annimg-stage.region-tool { cursor: crosshair; } + .annimg-stage img { + display: block; + max-width: 100%; + height: auto; + -webkit-user-drag: none; + user-select: none; + } + .annimg-overlay { + position: absolute; + inset: 0; + width: 100%; + height: 100%; + pointer-events: none; + } + .ann-region { + fill: color-mix(in srgb, var(--accent) 18%, transparent); + stroke: var(--accent); + stroke-width: 0.5; + vector-effect: non-scaling-stroke; + } + .ann-cross { + stroke: var(--accent); + stroke-width: 1; + stroke-dasharray: 2 2; + vector-effect: non-scaling-stroke; + opacity: 0.7; + } + .ann-point { + fill: var(--accent); + stroke: var(--accent-fg, #fff); + stroke-width: 0.5; + vector-effect: non-scaling-stroke; + } + .annimg-readout { + display: flex; + flex-wrap: wrap; + gap: 6px 12px; + font-size: 11.5px; + color: var(--muted); + } + .annimg-readout code { font-size: 11.5px; } + .annimg-empty { font-size: 11.5px; color: var(--muted); } + /* --- image grid --- */ .image-grid { display: grid; diff --git a/docs/skill.md b/docs/skill.md index a8b9a89..e347d92 100644 --- a/docs/skill.md +++ b/docs/skill.md @@ -73,6 +73,7 @@ Skip the dialog for content the user reads, doesn't answer: | Pick one of *many* images (e.g. 12 logo variants) | `form` with `image_grid` | | Per-item verdict on a *batch* of images/videos ("approve/revise/skip each") | `gallery` | | Pick one of 2–3 full variants shown side by side (drafts, headlines, before/after) | `compare` | +| Mark *where* on an image (point / region) | `form` with `annotated_image` | | Single free-text answer | just ask in chat | | More than 8 fields | split into multiple `form` calls; do not cram one dialog | @@ -221,6 +222,62 @@ Read-only — sits between input fields to give layout context, like - Free-form HTML / markdown inside `content` — content is plain text, rendered monospace; everything else is intentionally ignored. +## Mark a point or region on an image: `annotated_image` + +When the answer you need is *spatial* — "where should the logo go?", "which +part do I crop?", "point at the bug in this screenshot" — words are a poor +carrier. Show the image and let the user mark it directly. + +Spec: +`{kind: "annotated_image", name, src, label?, alt?, mode?, max_height?, required?, default?}`. + +- `src` — same resolution rules as `image` (absolute / `~/` local path, + `http(s)://` URL, or `data:` URL). See [Image sources](#image-sources-src--thumbnail). +- `mode` — what the user can mark: + - `"point"` (default) — click to drop a single crosshair marker. + - `"region"` — drag to draw a rectangle. + - `"both"` — a Point/Region toggle appears; the user can set a point + *and* a region (both are returned). +- `default` — optionally seed `{point?: {x, y}, region?: {x, y, w, h}}` in + normalized units to pre-place a marker the user then nudges. +- `required` — the submit action stays disabled until the user has marked + the annotation the mode calls for. + +**Result** (under the field `name`): + +``` +{ + "point": {"x": 0.42, "y": 0.31} | null, + "region": {"x": 0.10, "y": 0.20, "w": 0.30, "h": 0.25} | null, + "natural": {"width": 1920, "height": 1080} | null +} +``` + +All coordinates are **normalized 0..1** relative to the image — resolution +independent, so they survive the image being displayed at any size. `region` +is top-left `x,y` plus `w,h`. Multiply by `natural` (the image's intrinsic +pixel size, filled in once it loads) to get pixel coordinates: +`px = point.x * natural.width`. + +``` +{ + "kind": "annotated_image", + "name": "logo_spot", + "label": "Where should the logo sit?", + "src": "~/renders/hero.png", + "mode": "point" +} +``` + +**Anti-patterns:** + +- Asking "top-left or bottom-right?" in a `select` when the honest answer + is a spot on the image — that's exactly what this field is for. +- Using it for *picking one image out of many* — that's `image_grid` (this + field annotates a **single** image). +- Expecting pixel coordinates in the result without reading `natural` — the + raw `x/y/w/h` are fractions, not pixels. + ## Inline-context fields: `markdown`, `image`, `static_text` These don't ask anything — they sit between input fields to give context diff --git a/python/src/aiui_mcp/server.py b/python/src/aiui_mcp/server.py index 2881bb9..eda2203 100644 --- a/python/src/aiui_mcp/server.py +++ b/python/src/aiui_mcp/server.py @@ -867,6 +867,7 @@ async def form( - static_text: {kind, text, tone?: "info"|"warn"|"muted"} — display only - markdown: {kind, text} — read-only Markdown block; only as inline context for following inputs in the same form, NOT as a standalone display tool. - image: {kind, src, label?, alt?, max_height?} — read-only image. `src` accepts an absolute / `~/` local path (read on YOUR host), an `http(s)://` URL (fetched on the Mac), or a `data:` URL. Use for visual confirmation of agent-generated previews. + - annotated_image: {kind, name, src, label?, alt?, mode?, max_height?, required?, default?} — let the user MARK a spot on an image (logo placement, crop hint, bug location). `src` follows the same rules as `image`. `mode` ∈ {"point" (click one marker, default), "region" (drag a rectangle), "both" (user flips a Point/Region tool)}. `default` may seed `{point?: {x, y}, region?: {x, y, w, h}}` in normalized units. Result under `name`: {point: {x, y} | null, region: {x, y, w, h} | null, natural: {width, height} | null} — all coordinates normalized 0..1; multiply by `natural` for pixels. - mermaid: {kind, source, label?, max_height?} — read-only Mermaid diagram (flowchart, sequence, state, gantt, mindmap, …). `source` is a Mermaid-DSL string. Use this instead of ASCII / box-drawing art when you'd otherwise sketch a diagram in chat — aiui renders to SVG and DOMPurify-sanitises before display. - wireframe: {kind, panels: [{title?, content?, col_span?, row_span?, tone?}], columns?, gap?, label?, max_height?} — read-only UI-layout mockup. Real CSS-Grid panels with optional header (`title`) and multi-line monospace body (`content`, escape `\n`). `tone` ∈ {"default","muted","highlight"}. Use this for *UI-layouts* (dashboard tiles, hardware-UI panels, login screens, anything with fixed-position boxes-and-labels) instead of ASCII boxes-and-pipes — `mermaid` is for *diagrams* (graphs, sequence/state, gantt). Wireframe complements it for the layout class. - image_grid: {kind, name, label?, images: [{value, src, label?}], multi_select?, columns?, default_selected?, required?}