Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added
- Windows-only `native_dialog_snapshot` and `native_dialog_interact` tools use Microsoft UI Automation to inspect and control native dialogs owned by the connected Tauri application, including multi-file Open, folder selection, navigation controls, nested ownership chains, and Save overwrite confirmations.

## [0.12.0] - 2026-07-05

### Added
Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,8 @@ Just type the command in your AI assistant to start a guided workflow.
| `read_logs` | Read console, Android, iOS, or system logs |
| `webview_interact` | Click, scroll, swipe, focus, long-press |
| `webview_screenshot` | Capture webview screenshots |
| `native_dialog_snapshot` | Inspect app-owned native Windows dialog chains and navigation controls |
| `native_dialog_interact` | Invoke, set single/multiple paths, or select native Windows dialog controls |
| `webview_keyboard` | Type text or send key events |
| `webview_wait_for` | Wait for elements, text, or events |
| `webview_get_styles` | Get computed CSS styles |
Expand All @@ -135,6 +137,8 @@ Just type the command in your AI assistant to start a guided workflow.

> **Multi-Window Support**: All webview tools accept an optional `windowId` parameter to target specific windows. Use `manage_window` with `action: "list"` to discover available windows.

> **Native Windows dialogs**: `native_dialog_*` tools require an interactive desktop and expose only same-process dialogs whose owner chain leads to the targeted Tauri window. They support single/multi-file Open, Save, folder selection, nested confirmations, and advertised navigation controls, but not toast notifications or other applications.

</details>

<details>
Expand Down
5 changes: 3 additions & 2 deletions docs/api/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: API Reference - What Your AI Can Do
description: Explore 20 powerful MCP tools for Tauri development including setup, mobile development, UI automation, and IPC debugging.
description: Explore 22 powerful MCP tools for Tauri development including setup, mobile development, UI automation, and IPC debugging.
head:
- - meta
- name: keywords
Expand All @@ -13,7 +13,7 @@ import { Wrench, Smartphone, Target, Zap, Rocket, Bug, FlaskConical, Activity, M

# What Your AI Can Do

Once configured, your AI assistant has **20 powerful tools** plus **slash commands** to help you build, test, and debug your Tauri application. Just ask in natural language!
Once configured, your AI assistant has **22 powerful tools** plus **slash commands** to help you build, test, and debug your Tauri application. Just ask in natural language!

## <Wrench :size="24" :stroke-width="2" class="heading-icon" /> Setup & Configuration

Expand Down Expand Up @@ -61,6 +61,7 @@ Your AI can interact with your app's interface just like a user would.
- Verify UI elements exist and work correctly
- Check visual appearance with screenshots
- Visually select elements with the element picker
- Inspect and interact with app-owned native dialogs on Windows

**Debugging**
- See console logs and errors
Expand Down
75 changes: 74 additions & 1 deletion docs/api/ui-automation.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ head:

# UI Automation

Control and automate your Tauri application's UI. These tools provide comprehensive automation capabilities for testing and interaction, working seamlessly across all platforms (Linux, Windows, macOS).
Control and automate your Tauri application's UI. Webview tools work across Linux, Windows, and macOS. Native dialog automation is Windows-only.

## Multi-Window Support

Expand Down Expand Up @@ -222,6 +222,79 @@ For real iOS/Android devices on the network:

**Note**: No external driver process required.

## Native Windows Dialogs

`native_dialog_snapshot` and `native_dialog_interact` automate native message, confirmation, single- or multi-file Open, folder selection, and Save dialogs. These windows are outside the webview DOM, so `webview_interact`, `webview_keyboard`, and `webview_screenshot` cannot inspect or control them.

The tools require an active `driver_session` and an interactive Windows desktop. Discovery is restricted to visible dialogs in the connected Tauri process whose bounded owner chain leads back to the targeted Tauri window. This includes nested prompts such as a Save overwrite confirmation without exposing desktop-wide automation or dialogs from other applications. Windows toast notifications are not dialogs and are not supported.

### native_dialog_snapshot

Returns a bounded semantic snapshot containing control types, automation IDs, names, semantic roles, supported UI Automation patterns, owner depth, parent-dialog references, and opaque `elementRef` values. File-dialog snapshots include actionable navigation controls, the current-location control, navigation-tree items, and selectable file-system entries. Use `windowId` and `appIdentifier` with the same targeting behavior as webview tools. `timeoutMs` accepts 100–10000 milliseconds.

```javascript
{
"tool": "native_dialog_snapshot",
"windowId": "main",
"minOwnerDepth": 1,
"timeoutMs": 2000
}
```

`minOwnerDepth` defaults to `1`, where the Tauri window directly owns the dialog. Set it to `2` to wait for a prompt owned by another app-owned dialog, such as an overwrite confirmation. Nested dialogs are returned before their parents.

Controls advertise one or more supported actions:

- `invoke` — `InvokePattern`, typically buttons such as accept, cancel, yes, or no
- `setValue` — `ValuePattern`, used for a complete absolute filename/path
- `setPaths` — `ValuePattern`, used for up to 100 complete absolute existing file paths in a multi-select Open dialog
- `select` — `SelectionItemPattern`, where the dialog exposes a selectable control

Semantic roles are derived from automation IDs and control patterns rather than localized button text. Names and dialog text are returned to the caller for inspection but are not written to bridge logs.

### native_dialog_interact

Apply one advertised action to an `elementRef` from the latest snapshot:

```javascript
{
"tool": "native_dialog_interact",
"action": "invoke",
"elementRef": "native_..."
}
```

For Open or Save dialogs, set the filename field to a complete absolute path, then invoke the accept button:

```javascript
{
"tool": "native_dialog_interact",
"action": "setValue",
"elementRef": "native_...",
"value": "C:\\tmp\\fixture.txt"
}
```

For a multi-file Open dialog, use the filename field's advertised `setPaths` action and then invoke accept:

```javascript
{
"tool": "native_dialog_interact",
"action": "setPaths",
"elementRef": "native_...",
"paths": [
"C:\\tmp\\first.txt",
"C:\\tmp\\second.txt"
]
}
```

For a folder picker, use `setValue` with a complete absolute directory path and invoke accept. To respond to a Save overwrite prompt, invoke Save, take a new snapshot with `minOwnerDepth: 2`, and invoke the confirmation's advertised affirmative action. Back, forward, up, address/current-location, navigation-tree, and file-system entry controls are included when Windows exposes their corresponding UI Automation patterns.

Element references are session-bound, expire after 30 seconds, and are otherwise ephemeral. A new snapshot replaces the previous reference set; invoking or selecting a control invalidates references for that dialog, while `setValue` and `setPaths` preserve them so the previously discovered accept control can be invoked. If the dialog changes or closes, the tool returns a stale-reference error and the caller must take another snapshot.

Supported scope includes single- and multi-file Open, Save, folder selection, overwrite confirmation, message acknowledgement, ask/confirm responses, absolute-path entry, cancellation, nested app-owned dialogs, and navigation controls that expose the patterns above. Arbitrary custom controls without UI Automation patterns, permission prompts outside the app's ownership chain, notification/toast automation, image recognition, and coordinate-based native control remain out of scope.

## webview_find_element

Find UI elements using CSS, XPath, or text selectors.
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -286,7 +286,7 @@ Some MCP clients don't support slash commands yet. See the [manual prompt instru

## Next Steps

- **[API Reference](/api/)** — Learn about all 20 available tools
- **[API Reference](/api/)** — Learn about all 22 available tools
- **[IPC & Plugin Tools](/api/ipc-plugin)** — Debug your app's IPC layer
- **[UI Automation](/api/ui-automation)** — Automate webview interactions

Expand Down
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ The server provides **slash commands** ([prompts](https://modelcontextprotocol.i

[Learn more about prompts →](/api/prompts)

## 21 Powerful Tools
## 22 Powerful Tools

The server exposes tools across 4 categories:

Expand All @@ -190,7 +190,7 @@ The server exposes tools across 4 categories:
</div>
<div class="tool-category">
<Target :size="20" :stroke-width="2" class="category-icon" />
<strong>UI Automation & WebView</strong> (14 tools) - Gestures, screenshots, DOM snapshots, JS execution, element picking, logs
<strong>UI Automation & WebView</strong> (15 tools) - Webview and native Windows dialog automation, screenshots, DOM snapshots, element picking, logs
</div>
<div class="tool-category">
<Zap :size="20" :stroke-width="2" class="category-icon" />
Expand Down
3 changes: 3 additions & 0 deletions packages/mcp-server/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added
- Add `native_dialog_snapshot` and `native_dialog_interact` MCP tools for semantic Windows native-dialog automation, including multiple paths, folder pickers, navigation controls, and nested confirmations.

## [0.12.0] - 2026-07-05

### Added
Expand Down
4 changes: 3 additions & 1 deletion packages/mcp-server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ await driver_session({ action: "stop", appIdentifier: 9223 })
await driver_session({ action: "stop" })
```

## Available Tools (20 total)
## Available Tools (22 total)

### Setup & Configuration

Expand All @@ -115,6 +115,8 @@ await driver_session({ action: "stop" })
| `read_logs` | Read console, Android, iOS, or system logs |
| `webview_interact` | Click, scroll, swipe, focus, long-press |
| `webview_screenshot` | Capture webview screenshots (JPEG default) |
| `native_dialog_snapshot` | Inspect app-owned native Windows dialog chains and navigation controls |
| `native_dialog_interact` | Invoke, set paths, or select native Windows dialog controls |
| `webview_keyboard` | Type text or send key events |
| `webview_wait_for` | Wait for elements, text, or events |
| `webview_get_styles` | Get computed CSS styles |
Expand Down
122 changes: 122 additions & 0 deletions packages/mcp-server/src/driver/native-dialog.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
import { randomUUID } from 'node:crypto';
import { z } from 'zod';

import { ensureSessionAndConnect, type PluginClient } from './plugin-client.js';

const TargetSchema = {
windowId: z.string().optional().describe('Tauri window label whose owned native dialog chain should be targeted'),
appIdentifier: z.union([ z.string(), z.number() ]).optional().describe(
'App identifier (port or bundle ID) when multiple Tauri apps are connected'
),
timeoutMs: z.number().int().min(100).max(10000).default(2000).describe(
'Bounded time to wait for the native operation, from 100 to 10000 milliseconds'
),
} as const;

/** Schema for discovering native dialogs owned by the targeted Tauri window. */
export const NativeDialogSnapshotSchema = z.object({
...TargetSchema,
minOwnerDepth: z.number().int().min(1).max(8).default(1).describe(
'Minimum owner-chain depth to return: 1 for direct dialogs, 2 for confirmations owned by another dialog'
),
});

const InteractionTargetSchema = {
elementRef: z.string().min(1).describe('Opaque elementRef from the latest native_dialog_snapshot'),
...TargetSchema,
} as const;

/**
* Schema for invoking UI Automation patterns on a discovered native dialog element.
*/
export const NativeDialogInteractSchema = z.discriminatedUnion('action', [
z.object({
action: z.literal('invoke').describe('Invoke a button through UI Automation InvokePattern'),
...InteractionTargetSchema,
}),
z.object({
action: z.literal('setValue').describe('Set an editable filename/path through UI Automation ValuePattern'),
value: z.string().min(1).max(32767).describe('Complete absolute filename/path; the value is never logged'),
...InteractionTargetSchema,
}),
z.object({
action: z.literal('setPaths').describe('Set one or more existing files in a multi-select Open dialog'),
paths: z.array(z.string().min(1).max(32767)).min(1).max(100).describe(
'Complete absolute file paths for a multi-file Open dialog; path values are never logged'
),
...InteractionTargetSchema,
}),
z.object({
action: z.literal('select').describe('Select a control through UI Automation SelectionItemPattern'),
...InteractionTargetSchema,
}),
]);

export type NativeDialogSnapshotOptions = z.input<typeof NativeDialogSnapshotSchema>;
export type NativeDialogInteractOptions = z.input<typeof NativeDialogInteractSchema>;

const sessionScopes = new WeakMap<PluginClient, string>();

function getSessionScope(client: PluginClient): string {
const existing = sessionScopes.get(client);

if (existing) {
return existing;
}

const scope = randomUUID();

sessionScopes.set(client, scope);
return scope;
}

function resultAsJSON(data: unknown): string {
return JSON.stringify(data, null, 2);
}

/** Discover bounded semantic UI Automation controls in native Windows dialogs. */
export async function snapshotNativeDialog(options: NativeDialogSnapshotOptions): Promise<string> {
const parsed = NativeDialogSnapshotSchema.parse(options),
client = await ensureSessionAndConnect(parsed.appIdentifier);

const response = await client.sendCommand({
command: 'native_dialog_snapshot',
args: {
windowLabel: parsed.windowId,
timeoutMs: parsed.timeoutMs,
scopeId: getSessionScope(client),
minOwnerDepth: parsed.minOwnerDepth,
},
}, parsed.timeoutMs + 1500);

if (!response.success) {
throw new Error(response.error || 'Native dialog snapshot failed');
}

return resultAsJSON(response.data);
}

/** Interact with a native dialog element using its advertised UI Automation pattern. */
export async function interactWithNativeDialog(options: NativeDialogInteractOptions): Promise<string> {
const parsed = NativeDialogInteractSchema.parse(options),
client = await ensureSessionAndConnect(parsed.appIdentifier);

const response = await client.sendCommand({
command: 'native_dialog_interact',
args: {
windowLabel: parsed.windowId,
timeoutMs: parsed.timeoutMs,
scopeId: getSessionScope(client),
elementRef: parsed.elementRef,
action: parsed.action,
value: parsed.action === 'setValue' ? parsed.value : undefined,
paths: parsed.action === 'setPaths' ? parsed.paths : undefined,
},
}, parsed.timeoutMs + 1500);

if (!response.success) {
throw new Error(response.error || 'Native dialog interaction failed');
}

return resultAsJSON(response.data);
}
4 changes: 3 additions & 1 deletion packages/mcp-server/src/driver/protocol.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@ export type PluginCommandType =
| 'stop_ipc_monitor'
| 'get_ipc_events'
| 'execute_js'
| 'capture_native_screenshot';
| 'capture_native_screenshot'
| 'native_dialog_snapshot'
| 'native_dialog_interact';

/** Request message sent from MCP server to Tauri plugin */
export interface PluginRequest {
Expand Down
51 changes: 51 additions & 0 deletions packages/mcp-server/src/tools-registry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ import {
selectElement, getPointedElement,
SelectElementSchema, GetPointedElementSchema,
} from './driver/element-picker.js';
import {
interactWithNativeDialog, snapshotNativeDialog,
NativeDialogInteractSchema, NativeDialogSnapshotSchema,
} from './driver/native-dialog.js';
import { SETUP_INSTRUCTIONS } from './constants.js';

/**
Expand Down Expand Up @@ -287,6 +291,53 @@ export const TOOLS: ToolDefinition[] = [
return result.content;
},
},
{
name: 'native_dialog_snapshot',
description:
'[Windows Tauri Apps Only] Discover native message, confirmation, Open, or Save dialogs ' +
'in the ownership chain of a targeted Tauri window. Returns a bounded semantic UI Automation snapshot ' +
'with ownership depth, navigation and file-system controls, automation IDs, semantic roles, supported patterns, ' +
'and ephemeral elementRef values. Set minOwnerDepth to 2 to wait for a child confirmation such as Save overwrite. ' +
'Requires an active driver_session and an interactive Windows desktop. ' +
MULTI_APP_DESC + ' Native dialogs are outside the webview DOM; Windows toast notifications are not supported.',
category: TOOL_CATEGORIES.UI_AUTOMATION,
schema: NativeDialogSnapshotSchema,
annotations: {
title: 'Inspect Native Windows Dialog',
readOnlyHint: true,
destructiveHint: false,
openWorldHint: false,
},
handler: async (args) => {
const parsed = NativeDialogSnapshotSchema.parse(args);

return await snapshotNativeDialog(parsed);
},
},
{
name: 'native_dialog_interact',
description:
'[Windows Tauri Apps Only] Interact with an elementRef from the latest native_dialog_snapshot ' +
'using UI Automation InvokePattern, ValuePattern, or SelectionItemPattern. Use invoke for dialog buttons, ' +
'setValue with a complete absolute file or folder path, setPaths with complete absolute existing file paths ' +
'for multi-select Open dialogs, and select for advertised navigation or file-system controls. ' +
'References are session-bound, expire after 30 seconds, and are otherwise ephemeral; ' +
'take a new snapshot after a stale-reference error. ' +
'Requires an active driver_session and an interactive Windows desktop. ' + MULTI_APP_DESC,
category: TOOL_CATEGORIES.UI_AUTOMATION,
schema: NativeDialogInteractSchema,
annotations: {
title: 'Interact with Native Windows Dialog',
readOnlyHint: false,
destructiveHint: false,
openWorldHint: false,
},
handler: async (args) => {
const parsed = NativeDialogInteractSchema.parse(args);

return await interactWithNativeDialog(parsed);
},
},
{
name: 'webview_keyboard',
description:
Expand Down
Loading