Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---
name: flutter-mcp-cli-runtime-validation
description: Run Flutter MCP runtime validation from CLI in two steps (launch app, then run validate-runtime), including toolkit-extension gating, screenshot/layout capture, app error collection, optional reload verification, and retry handling for transient first-connect failures.
---
> Calls in this skill run via the `flutter-mcp-toolkit` CLI binary:
> flutter-mcp-toolkit exec --name <tool> --args '<json>'
> Output is JSON on stdout. Errors come as `{"error":{"code":..., "message":..., "recovery":...}}`.
> Throughout this skill, calls are written as `tap_widget(selector: "...")` — translate to the CLI form.
> If the binary isn't on PATH, see `flutter-mcp-toolkit-setup`.

# Flutter MCP CLI Runtime Validation

Use this skill when you need agent-style runtime validation through `flutter-mcp-toolkit` with minimal operator steps.

## Two-Step Flow

1. Launch the Flutter app in debug mode.
2. Run one CLI command:

```bash
dart run mcp_server_dart/bin/flutter_mcp_toolkit.dart --save-images --output-dir .flutter_mcp/runtime_validation validate-runtime \
--target ws://127.0.0.1:8181/<token>/ws \
--timeout-ms 10000 \
--post-reload-delay-ms 500 \
--after-reload
```

Optional skill install in the same command:

```bash
dart run mcp_server_dart/bin/flutter_mcp_toolkit.dart validate-runtime \
--target ws://127.0.0.1:8181/<token>/ws \
--install-skill
```

Permission behavior for this flow:

- `validate-runtime` stays read/write only for visual capture and defaults to `auto_request_once`.
- `doctor` remains read-only.
- On macOS, Screen Recording permission belongs to the host process running `flutter-mcp-toolkit`.
- On web: `desktop_window` uses macOS ScreenCaptureKit then Chrome CDP (`Page.captureScreenshot`); Linux/Windows use CDP when remote debugging is reachable. Pass `--web-browser-debugging-port` if discovery fails. With platform views detected, `validate-runtime` does not fall back to `flutter_layer` after a successful `desktop_window` capture. Check `captureHints.weakSignalsDetected` for `Texture`-only apps.
- Executor recovery retries host capture once (`desktopCaptureRetried` in screenshot payloads). When `captureHints.platformViewsDetected` is true, validate-runtime does not fall back to `flutter_layer`. Otherwise it may retry once with `flutter_layer` after a failed host capture.
- You may pass the VM URI as global `--vm-service-uri` instead of `validate-runtime --target` when only one URI is needed.

## What `validate-runtime` Must Prove

- Doctor preflight passes critical checks.
- Required toolkit extensions exist:
- `ext.mcp.toolkit.app_errors`
- `ext.mcp.toolkit.view_details`
- `ext.mcp.toolkit.view_screenshots`
- `ext.mcp.toolkit.inspect_widget_at_point`
- Screenshot capture works.
- View details (layout metadata) are available.
- App errors are retrievable.
- If `--after-reload` is enabled, post-reload screenshot also works.

## Output Handling

- Use `data.summary` as pass/fail status for automation.
- Use `data.summary.capturePlatformViewsDetected` and `captureFocusAttempted` for platform-view routing.
- Use `data.summary.captureFallbackUsed` to see whether a `flutter_layer` retry ran (skipped when platform views were detected).
- Use `data.steps` for per-step evidence and retries.
- Use `data.doctor.checks` to explain setup blockers.
- Use `data.summary.screenshotFiles` for saved screenshot paths when `--save-images` is enabled.
- When `--save-images` is enabled, read screenshot file URLs from step data.
- For visual debugging reports, also run:
- `exec --name capture_ui_snapshot --args '{"errorsCount":4,"compress":true,"includeViewDetails":true,"includeErrors":true}'`
- `exec --name inspect_widget_at_point --args '{"x":<int>,"y":<int>}'`

## Failure Rules

- If toolkit extensions are missing, stop and report instrumentation gap with exact fix:
- add `mcp_toolkit` to app dependencies
- ensure `MCPToolkitBinding.instance.bootstrapFlutter(...)` or equivalent manual initialization runs before `runApp`
- hot restart or rerun the app
- If first explicit URI connect fails, retry is automatic for retryable connection errors.
- If screenshots are blank, verify app window is visible and retry.
- If macOS visual capture is denied, use:
- `dart run mcp_server_dart/bin/flutter_mcp_toolkit.dart permissions status`
- `dart run mcp_server_dart/bin/flutter_mcp_toolkit.dart permissions request`
- `dart run mcp_server_dart/bin/flutter_mcp_toolkit.dart permissions open-settings`
- If app cannot be instrumented, do not claim screenshot/layout/error inspection success.

## Visual QA + Source Mapping Rules

- Always compare before/after screenshot evidence around changes.
- For each reported visual issue, provide coordinate + `inspect_widget_at_point` output.
- Map defects to source using `get_app_errors` top stack frame (`file`, `line`, `column`) when available.
- Do not use `debug_dump_*` unless explicitly requested.

## Challenge Cases (Always Call Out Explicitly)

- No running debug app: `doctor` critical failure on `vm_target_reachable`; request app launch before continuing.
- Wrong target URI/token: treat as connection mismatch and retry with exact `app.debugPort.wsUri`.
- Toolkit added but still missing extensions: hot reload is often insufficient, require hot restart/full rerun.
- Non-modifiable app (cannot add toolkit): report inspection as unavailable instead of guessing.
Original file line number Diff line number Diff line change
@@ -0,0 +1,239 @@
---
name: flutter-mcp-toolkit-control
description: Drive a running Flutter app — tap, scroll, type, fill forms, hot-reload, navigate. Use when you need to interact with the UI.
---
> Calls in this skill run via the `flutter-mcp-toolkit` CLI binary:
> flutter-mcp-toolkit exec --name <tool> --args '<json>'
> Output is JSON on stdout. Errors come as `{"error":{"code":..., "message":..., "recovery":...}}`.
> Throughout this skill, calls are written as `tap_widget(selector: "...")` — translate to the CLI form.
> If the binary isn't on PATH, see `flutter-mcp-toolkit-setup`.

## When to use

Use this skill when you need to drive a running Flutter app as a user would:
- Tap buttons, icons, list items, or any interactive widget.
- Type text into fields, submit forms, clear inputs.
- Scroll or swipe to reveal off-screen content.
- Navigate between routes programmatically (push, pop, popUntil).
- Dismiss dialogs and bottom sheets.
- Press keyboard keys (Enter, Escape, Tab, arrows, ASCII chars).
- Hot-reload or hot-restart after editing Dart source files.
- Combine reload + screenshot + semantics in one round-trip for fast iteration.

## Selectors

Every interaction tool targets a widget by **ref** — a short string like `"s_0"` returned by `semantic_snapshot`. There is no by-text or by-type selector syntax on the tool itself. The workflow is: call `semantic_snapshot`, scan the returned nodes, find the right ref, then pass it.

Snapshot node fields to filter on:

| Want to find | Scan field | Example value |
|---|---|---|
| By visible label / text | `label` | `"Login"` |
| By value or hint | `value` / `hint` | `"user@example.com"` |
| By tooltip | `tooltip` | `"Close"` |
| By widget key | `key` | `"[<'submitBtn'>]"` |
| By semantic role / type | `flags` or `actions` | `["tap"]` |

Example — find the "Login" button ref:
```
semantic_snapshot()
→ nodes: [{ref:"s_0", label:"Login", actions:["tap"]}, ...]
tap_widget(ref: "s_0")
```

Pass `snapshotId` (from the snapshot response) to any interaction call. If the tree has changed, the call returns `stale_snapshot` with both IDs so you know to re-snapshot. Refs are only valid against the most recent snapshot.

## Recipes

### Tap a widget by text
```
semantic_snapshot()
→ find node where label == "Submit" → ref "s_3"
tap_widget(ref: "s_3", snapshotId: <id>)
```

### Fill a login form
```
semantic_snapshot() → email ref "s_1", password ref "s_2"
fill_form(fields: [{ref:"s_1", text:"user@example.com"}, {ref:"s_2", text:"secret"}], snapshotId: <id>)
→ one round-trip; stops on first failure
```

### Scroll to find an item
```
scroll(direction: "down", distance: 300)
semantic_snapshot() → item now visible → ref "s_5"
tap_widget(ref: "s_5")
```

### Wait for a widget to appear
```
wait_for(predicate: {kind: "text", text: "Welcome"}, timeoutMs: 8000)
→ returns fresh snapshot when text appears
tap_widget(ref: <ref from wait_for snapshot>)
```

### Navigate to a route
```
navigate(action: "push", route: "/settings", arguments: {tab: "account"})
semantic_snapshot() → fresh refs in the new screen
```

### Hot reload after a code change
```
hot_reload_and_capture()
→ screenshot + semantic snapshot + errors in one call
```

### Press the back hardware button

`press_key` has no `Back` key. Use `navigate(action: "pop")` for Navigator pop; `handle_dialog(action: "dismiss")` for dialogs; `press_key(key: "Escape")` on desktop.

```
navigate(action: "pop")
```

## Tool reference

### tap_widget
Tap a widget by ref. `ref` • string • required. `snapshotId` • integer • optional. `connection` • object • optional.
```json
{"name": "tap_widget", "arguments": {"ref": "s_3", "snapshotId": 7}}
```
Returns: `{"via": "semantic_action", "ref": "s_3"}` — Failures: `stale_snapshot`, `ref_not_found`

### long_press
Long-press a widget by ref. `ref` • string • required. `snapshotId` • integer • optional. `connection` • object • optional.
```json
{"name": "long_press", "arguments": {"ref": "s_2"}}
```
Returns: `{"via": "semantic_action"}` — Failures: `stale_snapshot`, `ref_not_found`

### enter_text
Enter text into a text field; taps to focus before typing. `ref` • string • required. `text` • string • required. `snapshotId` • integer • optional. `connection` • object • optional.
```json
{"name": "enter_text", "arguments": {"ref": "s_1", "text": "hello@example.com"}}
```
Returns: `{"via": "editable_state"}` — Failures: `stale_snapshot`, `ref_not_found`

### fill_form
Batch text entry: fills multiple fields in one call. Stops on first failure. `snapshotId` validated on first field only. `fields` • array of `{ref, text}` • required. `snapshotId` • integer • optional. `connection` • object • optional.
```json
{"name": "fill_form", "arguments": {"fields": [{"ref":"s_1","text":"user"},{"ref":"s_2","text":"pass"}], "snapshotId": 5}}
```
Returns: `{"filled": 2}` — Failures: `stale_snapshot`, `ref_not_found`

### scroll
Scroll to reveal content. `"down"` reveals content below (finger swipes up). `direction` • string • required (`up|down|left|right`). `ref` • string • optional (falls back to screen center). `distance` • number • optional • default 300. `snapshotId` • integer • optional. `connection` • object • optional.
```json
{"name": "scroll", "arguments": {"direction": "down", "ref": "s_0", "distance": 500}}
```
Returns: `{"via": "semantic_action"}` — Failures: `ref_not_found`, `stale_snapshot`

### swipe
High-velocity fling. Same direction model as `scroll`. Always Tier 2 pointer events. `direction` • string • required. `ref` • string • optional. `distance` • number • optional • default 300. `snapshotId` • integer • optional. `connection` • object • optional.
```json
{"name": "swipe", "arguments": {"direction": "left", "ref": "s_4"}}
```
Returns: `{"via": "pointer_events"}` — Failures: `ref_not_found`, `web_gesture_not_supported`

### drag
Drag from one widget to another. Always Tier 2. `fromRef` • string • required. `toRef` • string • required. `snapshotId` • integer • optional. `connection` • object • optional.
```json
{"name": "drag", "arguments": {"fromRef": "s_2", "toRef": "s_7"}}
```
Returns: `{"via": "pointer_events"}` — Failures: `ref_not_found`, `web_gesture_not_supported`

### hover
Synthesize a mouse hover. Desktop/web only — no hover concept on mobile. `ref` • string • required. `snapshotId` • integer • optional. `connection` • object • optional.
```json
{"name": "hover", "arguments": {"ref": "s_5"}}
```
Returns: `{"via": "pointer_events"}` — Failures: `ref_not_found`, platform error on mobile

### press_key
Synthesize key press (down+up). Accepted: `Enter Escape Tab Backspace Delete Space ArrowUp ArrowDown ArrowLeft ArrowRight` plus single ASCII (`a-z` `0-9`). `key` • string • required. `ctrl/shift/alt/meta` • boolean • optional • default false. `connection` • object • optional.
```json
{"name": "press_key", "arguments": {"key": "Enter"}}
```
Returns: `{"key": "Enter"}` — Failures: `unsupported_key`, `no_focus`

### wait_for
Wait for a UI predicate; returns fresh semantic snapshot. Predicates: `{kind:"text",text}` | `{kind:"noText",text}` | `{kind:"time",ms}` | `{kind:"stable",stableWindowMs}`. `predicate` • object • required. `timeoutMs` • integer • optional • default 5000 • max 30000. `connection` • object • optional.
```json
{"name": "wait_for", "arguments": {"predicate": {"kind": "text", "text": "Dashboard"}, "timeoutMs": 8000}}
```
Returns: fresh semantic snapshot — Failures: `timeout`, `invalid_predicate`

### navigate
Drive the registered Navigator. Requires `MCPToolkitBinding.instance.navigatorKey = key` in the app. `action` • string • required (`push|pop|popUntil`). `route` • string • required for push/popUntil. `arguments` • object • optional (for push). `connection` • object • optional.
```json
{"name": "navigate", "arguments": {"action": "push", "route": "/profile", "arguments": {"userId": "42"}}}
```
Returns: `{"action": "push", "route": "/profile"}` — Failures: `navigator_not_configured`, `route_not_found`

### handle_dialog
Dismiss the topmost popup/dialog route. Only `action: "dismiss"` supported. Requires `navigatorKey = key` on `MCPToolkitBinding.instance` in the app. `action` • string • required (must be `"dismiss"`). `connection` • object • optional.
```json
{"name": "handle_dialog", "arguments": {"action": "dismiss"}}
```
Returns: `{"dismissed": true}` — Failures: `navigator_not_configured`, `no_dialog`

### hot_reload_flutter
Hot reload the app. Preserves state. `force` • boolean • optional • default false (reload even without source changes). `connection` • object • optional.
```json
{"name": "hot_reload_flutter", "arguments": {}}
```
Returns: `"Hot reload completed"` + report JSON — Failures: `vm_not_connected`, `compilation_error`

### hot_restart_flutter
Full restart. App state not preserved. No required params. `connection` • object • optional.
```json
{"name": "hot_restart_flutter", "arguments": {}}
```
Returns: `{"report": {"type": "Success", "success": true}}` — Failures: `vm_not_connected`

### hot_reload_and_capture
Hot reload then capture screenshot + semantics + errors in one call. `compress` • boolean • default true. `includeSemantics` • boolean • default true. `includeErrors` • boolean • default true. `errorsCount` • integer • default 4. `connection` • object • optional.
```json
{"name": "hot_reload_and_capture", "arguments": {"includeErrors": true}}
```
Returns: screenshot (base64) + semantic snapshot + errors — Failures: `vm_not_connected`, `compilation_error`

## Patterns

### Always `wait_for` before `tap_widget` after navigation

After `navigate(action: "push")` the new route's widgets are not in the tree yet. Use `wait_for` with a text predicate to confirm the destination has rendered, then snapshot and act.

```
navigate(action: "push", route: "/checkout")
wait_for(predicate: {kind: "text", text: "Order Summary"}, timeoutMs: 5000)
semantic_snapshot() → tap target widgets
```

### Prefer `fill_form` over multiple `enter_text` calls

Each `enter_text` is a separate VM round-trip. `fill_form` sends all field/text pairs in one call; `snapshotId` is checked once (on the first field). For any form with 2+ fields, always prefer `fill_form`.

```
# Avoid: 2 round-trips
enter_text(ref: "s_1", text: "Alice")
enter_text(ref: "s_2", text: "secret")

# Prefer: 1 round-trip
fill_form(fields: [{ref: "s_1", text: "Alice"}, {ref: "s_2", text: "secret"}])
```

### After `hot_reload_*`, wait for the new tree before continuing

Hot reload completes asynchronously. Use `wait_for(predicate: {kind:"stable", stableWindowMs:300})` to confirm the tree has settled before calling `semantic_snapshot`. Or use `hot_reload_and_capture` which returns a post-reload snapshot directly.

```
hot_reload_flutter()
wait_for(predicate: {kind: "stable", stableWindowMs: 300})
semantic_snapshot() → interact with reloaded widgets

# Or in one call (preferred):
hot_reload_and_capture() → screenshot + semantics + errors already post-reload
```
Loading
Loading