fix(ui_type): use clipboard-based approach for non-US keyboard layouts

[davidlandais]

Summary

Fixes #43

This PR fixes the keyboard layout issue where ui_type sends incorrect characters on non-US keyboard layouts (AZERTY, QWERTZ, etc.).

Problem

The previous implementation used idb ui text which sends keystrokes assuming a US QWERTY keyboard layout. For example, typing mobiletest@example.com on a French AZERTY keyboard would result in ,obiletestàexa,ple:co, because:

• m key position → , on AZERTY
• @ (Shift+2) → different character on AZERTY

Solution

This fix uses a clipboard-based approach that is keyboard-layout independent:

1. Copy to clipboard: Use xcrun simctl pbcopy to copy text directly to the simulator's clipboard
2. Trigger paste: Use AppleScript to send Cmd+V to the Simulator app
3. Confirm paste: Auto-tap the "Paste" confirmation button that appears on iOS 16+ for clipboard access security

Internationalization

The paste button detection supports multiple locales:

• English: "Paste"
• French: "Coller"
• German: "Einsetzen"
• Spanish: "Pegar"
• Italian: "Incolla", "Inserisci"
• Dutch: "Plakken"
• Portuguese: "Colar"

Test plan

• Tested on macOS with French AZERTY keyboard
• Tested typing email addresses with special characters (@, .)
• Tested paste confirmation auto-tap on iOS 17 simulator
• Build passes with no TypeScript errors

Breaking changes

None. The function signature and return values remain unchanged.

[davidlandais]

Hi! 👋

I want to be transparent about this contribution. I'm not an iOS/Android development expert – I'm primarily a user who encountered this keyboard layout issue while using Claude (Opus 4.5) for my projects.

When I hit this bug, I asked the AI to analyze the problem and propose a fix. I cloned your repository, let Claude work on it, tested the solution on my setup (French AZERTY keyboard + iOS Simulator), and it worked for my use case.

Important caveats:

• This code was entirely AI-generated
• I cannot claim any expertise on the underlying iOS/idb internals
• I haven't tested edge cases or other keyboard layouts beyond French AZERTY

I'm absolutely not questioning your expertise or suggesting this is production-ready code. This PR is meant as a starting point or inspiration that might help accelerate your own implementation if you choose to address this issue.

Feel free to:

• Take what's useful and rewrite it properly
• Close this PR if the approach doesn't fit your vision
• Use it as a reference for your own solution

Thank you for maintaining this project. <3 Lovely from France.

[joshuayoes]

Hey, thanks for the pull request and the thoughtful comment! This approach adds a lot more complexity to their fairly simple command that I would like, but I also would like to make sure it's usable for non-US users. Ideally I'd like to fix it upstream in idb at facebook/idb#666, but I'm not sure they are accepting external contributions at the moment. I will need to some thinking on how I would like to address this gap. A few options off the top of my head:

1. Create a separate tool with this implementation.
2. Add a tool argument that uses this implementation for non-US inputs.
3. Find a way to upstream a fix into idb.
4. Do another implementation without idb for this tool.

I'm pretty swamped with some other professional projects and demands in my personal life at the moment, so I can't guarantee I will get to this quickly, but I would open to other community effort in this direction in the mean time.

[joshuayoes]

I got the chance to think about it a little bit more this weekend and with the help of Claude, I came up with an idea to keep the tool the same for simple use cases but include support for your approach for special characters and non US keyboards.

// Add near the top with other constants
const SAFE_TEXT_PATTERN = /^[a-zA-Z0-9 ]*$/;
const PASTE_BUTTON_LABELS = ["Paste", "Coller", "Einsetzen", "Pegar", "Incolla", "Plakken", "Colar"];

// Helper to find element in accessibility tree
function findElementByLabels(elements: any[], labels: string[]): any | null {
  for (const el of elements) {
    if (labels.includes(el.AXLabel)) return el;
    if (el.children) {
      const found = findElementByLabels(el.children, labels);
      if (found) return found;
    }
  }
  return null;
}

server.tool(
  "ui_type",
  "Input text into the iOS Simulator",
  {
    udid: z
      .string()
      .regex(UDID_REGEX)
      .optional()
      .describe("Udid of target, can also be set with the IDB_UDID env var"),
    text: z
      .string()
      .max(500)
      .regex(/^[\x20-\x7E]+$/)
      .describe("Text to input"),
  },
  { title: "UI Type", readOnlyHint: false, openWorldHint: true },
  async ({ udid, text }) => {
    try {
      const actualUdid = await getBootedDeviceId(udid);

      // Fast path: simple alphanumeric text uses idb directly
      if (SAFE_TEXT_PATTERN.test(text)) {
        const { stderr } = await idb("ui", "text", "--udid", actualUdid, "--", text);
        if (stderr) throw new Error(stderr);
        return {
          isError: false,
          content: [{ type: "text", text: "Typed successfully" }],
        };
      }

      // Clipboard path: handles @, #, special chars, non-US keyboards
      // 1. Copy to simulator clipboard via simctl
      await run("xcrun", ["simctl", "pbpaste", actualUdid, text]);
      
      // Note: simctl pbpaste reads from stdin, so we need spawn instead:
      await new Promise<void>((resolve, reject) => {
        const proc = spawn("xcrun", ["simctl", "pbcopy", actualUdid]);
        proc.stdin.write(text);
        proc.stdin.end();
        proc.on("close", (code) => (code === 0 ? resolve() : reject(new Error(`pbcopy failed with code ${code}`))));
        proc.on("error", reject);
      });

      // 2. Trigger Cmd+V (keycode 9 = 'v', modifier 1 = command)
      await idb("ui", "key", "--udid", actualUdid, "9", "--modifier", "1");

      // 3. Handle iOS 16+ paste confirmation with single describe-all call
      await new Promise((resolve) => setTimeout(resolve, 300));

      const { stdout } = await idb("ui", "describe-all", "--udid", actualUdid, "--json");
      const elements = JSON.parse(stdout);
      const pasteButton = findElementByLabels(elements, PASTE_BUTTON_LABELS);

      if (pasteButton) {
        const x = pasteButton.frame.x + pasteButton.frame.width / 2;
        const y = pasteButton.frame.y + pasteButton.frame.height / 2;
        await idb("ui", "tap", "--udid", actualUdid, String(x), String(y));
      }

      return {
        isError: false,
        content: [{ type: "text", text: "Typed successfully" }],
      };
    } catch (error) {
      return {
        isError: true,
        content: [
          {
            type: "text",
            text: errorWithTroubleshooting(
              `Error typing text into the iOS Simulator: ${toError(error).message}`
            ),
          },
        ],
      };
    }
  }
);

If you are willing to clean up the PR a little bit to get closer to this approach and verify that this works for you on your end, I would be willing to test on my end and consider adding it.

[davidlandais]

I've just read your message. My apologies. I have some time to try your code. I will come back to you. Ty

[davidlandais]

Hey @joshuayoes! 👋

Thanks for taking the time to think about this and for your detailed code suggestion! I had Claude (Opus 4.6) work on this under my supervision and validation, and we went through quite a research journey.

Testing your suggested approach

We started by implementing your hybrid approach (fast path with idb ui text for simple alphanumeric, clipboard path for special chars). Two issues came up:

1. idb ui key --modifier doesn't exist

The --modifier flag is not a valid argument for idb ui key in any released version of idb. We're on fb-idb 1.1.7 (pip), and the latest release is v1.1.8 (August 2022) — neither supports it:

idb ui key --udid <UDID> 9 --modifier 1
→ idb: error: unrecognized arguments: --modifier 1

We found PR #900 on facebook/idb which added individual modifier flags (--shift, --control, --option, --command) — not --modifier <int>. This code is on main but has never been released, and building from source fails due to gRPC/protobuf issues. So in practice, no one can use it today.

2. The fast path is broken on non-US keyboards

The SAFE_TEXT_PATTERN = /^[a-zA-Z0-9 ]*$/ fast path still uses idb ui text, which sends HID keycodes assuming QWERTY. On AZERTY, even simple letters are scrambled:

idb ui text "amazing jazz" → "q,qwing jqww"

a→q, m→,, z→w — all swapped per the AZERTY physical layout. The fast path only guards against special characters (@, #), but the fundamental problem is that all HID keycodes are QWERTY-mapped. This is the core of issue #666.

Our solution: AppleScript keystroke

After researching 9 different approaches (clipboard, XCUITest, Detox, Appium, simctl keyboard, Maestro, AppleScript, URL schemes, idb gRPC), we discovered that AppleScript's keystroke command sends Unicode characters directly, not HID keycodes. It is keyboard-layout-independent by design:

osascript -e 'tell application "System Events" to keystroke "amazing jazz"'
→ "amazing jazz" ✅ (correct on AZERTY!)

This is dramatically simpler than the clipboard approach:

	Clipboard (previous)	AppleScript keystroke (new)
Lines of code	~90	~8
Layout-independent	✅	✅
Special chars (@, #, !)	✅	✅
No iOS 16+ paste popup	❌ (needs handling)	✅
Simulates real typing	❌ (pastes at once)	✅ (char by char)
Requires Simulator focus	❌	✅

The only trade-off is that the Simulator must be the frontmost app, which seems acceptable for a local dev tool.

Test results on French AZERTY keyboard (iOS 16.4 Simulator)

Input	idb ui text (before)	AppleScript keystroke (after)
amazing jazz	q,qwing jqww ❌	amazing jazz ✅
test@example.com	N/A	test@example.com ✅
amazing jazz @Paris!	N/A	amazing jazz @Paris! ✅

The updated PR removes the clipboard approach entirely (-92 lines, +6 lines) and replaces it with a single osascript keystroke call. Build passes with no TypeScript errors.

Let me know what you think! Happy to adjust if needed.

[joshuayoes]

Thanks for the detailed pivot — really appreciate the research. I tried to reproduce your test matrix on iPhone 17 Pro / iOS 26.2 (macOS Darwin 25.3.0), granted Accessibility to the host terminal, Simulator frontmost, clean URL bar each run.

All three of your example cases run non-deterministically on my machine:

• amazing jazz → amazingjazz. + 45 trailing spaces (run 1), amazingjazz (run 2)
• test@example.com → 87 leading spaces + test@example.com (run 1), clean (run 2)
• amazing jazz @Paris! → amazingjazz@Paris! + 52 trailing spaces (run 1), amazingjazz@Pari s! (run 2)

pr44-trimmed.mp4

Video repro attached. Same input, different output each time — spacebar events from System Events keystroke are racing with the Simulator's keyboard handling and iOS autocorrect. This is a known reliability issue with AppleScript keystroke on bursts, and it's worse on faster machines because the keydowns queue up.

Also: Accessibility permission — required but undocumented. First call fails until the host process that spawned the MCP server is granted Accessibility under System Settings → Privacy & Security → Accessibility:

$ osascript -e 'tell application "System Events" to keystroke "x"'
36:49: execution error: System Events got an error:
osascript is not allowed to send keystrokes. (1002)

macOS TCC grants are per-app, so every MCP host (Cursor, Claude Desktop, Raycast, VS Code, various terminals…) needs its own grant. No README mention:

---

I think this will require some more research before this can be ready to merge.