feat: rewrite iOS Simulator integration with Swift helper + RN/native inspector bridges

[snowdamiz]

Problem

The current iOS Simulator sidebar (macOS-only) relies on a vendored idb-companion sidecar (client/src-tauri/build.rs fetches a pinned universal bundle on macOS), the gRPC idb client (commands/emulator/ios/idb_client.rs, behind the default ios-grpc feature), plus simctl + AppleScript HID fallbacks (xcrun.rs, cg_input.rs, input.rs). The result:

• Streaming/preview is janky (polled simctl screenshots + AppleScript fallback) and does not deliver a Radon-quality embedded preview.
• Tap/gesture forwarding is split across paths with inconsistent reliability.
• There is no element-level inspection: no hover-to-highlight, no click-to-jump-to-source for React Native, and no view-hierarchy surface for native Swift.
• We are pinned to idb-companion releases (last Facebook/idb release was August 2022; the repo is not archived but the cadence is irregular). Long term, leaning entirely on it is fragile, and we still re-implement parts ourselves anyway.

We want to use the iOS Simulator panel as a real IDE surface for users running React Native / Expo and native Swift projects from the workflow / agent / editor views.

Proposed solution

Replace the current ad-hoc stack with a dedicated macOS Swift helper binary that owns all low-level Simulator interaction, and add inspector bridges for both supported app frameworks. Communication with the Tauri Rust backend happens over a Unix domain socket (with stdio JSON-lines as a fallback) so we keep src-tauri clean and isolate any private-API usage.

Tauri App (Rust + Web Frontend)
├── Rust backend (orchestrator, Tauri commands, IPC)
├── Swift helper binary (macOS only)
│   ├── ScreenCaptureKit  → frame streaming
│   ├── SimulatorKit + Indigo HID → input injection
│   └── CoreSimulator + simctl → lifecycle / control
├── Metro Inspector bridge (React Native / Expo)
└── Accessibility (AX) bridge (native Swift)

Core technologies

Component	Technology	Status	Purpose
Lifecycle	xcrun simctl + CoreSimulator	Public, Apple-maintained	Boot / install / launch / shutdown
Frame capture	ScreenCaptureKit	Public, maintained (introduced WWDC22)	High-quality, low-latency capture of Simulator.app window
Input injection	SimulatorKit + Indigo HID over mach IPC	Private; protocol is the same one FBSimulatorControl reverse-engineered, still the de-facto standard	Reliable taps, gestures, keyboard
RN inspection	Metro inspector (/json/list + WebSocket on default port 8081)	Maintained (used by React Native DevTools / Expo)	Component tree, element-at-point, source mapping
Native Swift inspection	Accessibility (AXUIElement) APIs	Public, maintained	Element-at-point, label, role, AX identifier
IPC	Unix domain socket (JSON-lines fallback)	—	Rust ↔ Swift helper

Phased implementation

• Phase 1 — Foundation: smooth preview + reliable input (≈4–6 weeks). Stand up ios-sim-helper Swift binary (suggested layout: Package.swift, SimulatorManager.swift, CaptureManager.swift, InputManager.swift, InspectorManager.swift). Stream frames via ScreenCaptureKit, forward input via SimulatorKit/Indigo, expose Tauri commands start_preview(udid), send_tap(x,y), send_gesture(...), rotate_device, etc. Frontend renders frames into a <canvas> with proper coordinate mapping and an iPhone bezel overlay. Gate everything in Rust behind #[cfg(target_os = \"macos\")].
• Phase 2 — React Native / Expo deep integration (≈3–5 weeks). Auto-detect RN/Expo projects, ensure Metro is running, connect to its inspector WebSocket. Implement get_element_at(x, y) returning component name, props, bounds, source file + line. On mousemove / click over the canvas, draw a highlight overlay and jump to the corresponding location in the in-app editor. Stretch: component tree panel, props inspector, network inspector via Metro. This is the path that gets us close to Radon-IDE-level interactivity (hover for dimensions, click to open source) for RN/Expo.
• Phase 3 — Native Swift support (≈4–6 weeks). Use AXUIElement from the helper to answer get_accessible_element_at(x, y) (type, label, bounds, AX identifier). Frontend shows a highlight + small info panel; offer a "Search project for this AX identifier / label" affordance to bridge to the editor. Be explicit in the UI that source correlation is best-effort here. Optional / experimental: deeper view-hierarchy via private CoreSimulator debugging APIs (clearly flagged as fragile).
• Phase 4 — Polish (ongoing). Multi-simulator sessions, hot-reload detection + auto-restart, screen recording / replay, device-settings panel (location / appearance / language), graceful messaging on non-macOS builds.

Tauri layout

• client/src-tauri/src/commands/emulator/ios/: add helper.rs (Swift binary process management), inspector.rs (Metro WebSocket client + AX proxy), and reshape idb_client.rs / cg_input.rs / input.rs into the new flow (or behind a legacy feature flag during migration).
• build.rs: build / bundle / verify the Swift helper alongside the existing sidecar fetch logic; honor CADENCE_SKIP_SIDECAR_FETCH=1.
• Cargo features: introduce ios-helper (default on macOS), keep ios-grpc available during migration so we can fall back to idb-companion if the helper fails.
• Frontend: high-performance <canvas> preview, overlay layer (DOM or canvas) for highlights, hooks into the existing editor surface for jump-to-source.

Alternatives considered

• Stay on idb-companion only. Lowest risk and already wired up, but does not give us a smooth embedded preview or any element inspection — the original user pain points remain. Also leaves us exposed to idb's irregular release cadence (no formal release since Aug 2022).
• Pure-Rust ScreenCaptureKit + simctl, no Swift helper. ScreenCaptureKit + private SimulatorKit are Objective-C/Swift APIs; calling them from Rust via objc2 / FFI is doable but bleeds platform-specific complexity into src-tauri and makes private-API isolation harder. The helper-binary approach lets us version and replace the macOS-specific surface independently, and keeps Rust portable.
• Embed Simulator.app directly via window-reparenting / NSWindow capture. Brittle across macOS versions, conflicts with users running Simulator standalone, and offers no path to element inspection.
• WebDriverAgent / XCUITest for inspection. Heavier to set up per project, slower for hover-rate queries, and doesn't solve preview/streaming. Useful as a longer-term complement, not a foundation.

Success criteria

• Embedded iOS Simulator preview renders smoothly (target 30–60 FPS) inside the Cadence panel on macOS, including when Simulator.app is hidden / minimized.
• Tap, multi-touch gesture, hardware-keyboard, and rotation input round-trip through the Swift helper with parity-or-better reliability than today's simctl + AppleScript path.
• For React Native / Expo projects: hovering the preview highlights the element under cursor; clicking opens the corresponding source file + line in the in-app editor; component name, props, and bounds are surfaced in a side panel.
• For native Swift projects: hovering / clicking surfaces element type, label, bounds, and accessibility identifier; "Search project for this identifier" jumps into editor search. UI clearly communicates the reduced fidelity vs. RN.
• Non-macOS builds compile and surface a clear "iOS Simulator requires macOS" message rather than failing.
• New code paths covered by Vitest (frontend) and Rust integration tests (cargo test --manifest-path client/src-tauri/Cargo.toml), including a fake-helper harness so tests run without the real Simulator.
• Migration plan / feature flag in place so we can fall back to idb-companion while the Swift helper stabilizes; legacy code removed once parity is reached.
• Honest documentation in EMULATOR_SIDEBAR_PLAN.md (and README "iOS" section) describing the helper architecture, private-API risks, and the RN-vs-native fidelity gap.

Area

iOS simulator

[m13v]

polled simctl screenshots cap around 2-3 FPS once you account for PNG encode + IPC roundtrip. on the macOS capture side i went the other direction, SCStream from ScreenCaptureKit feeding CMSampleBuffer into VTCompressionSession with H.265, lands clean at 30 FPS hardware encoded with basically no jank. the Simulator window is just an NSWindow on the host, so SCStream can target it via SCContentFilter directly without going through simctl at all. swift helper as a daemon over UDS is the right call, just budget time for TCC, screen recording permission is sticky across user upgrades in weird ways and will bite you in support.

[snowdamiz]

polled simctl screenshots cap around 2-3 FPS once you account for PNG encode + IPC roundtrip. on the macOS capture side i went the other direction, SCStream from ScreenCaptureKit feeding CMSampleBuffer into VTCompressionSession with H.265, lands clean at 30 FPS hardware encoded with basically no jank. the Simulator window is just an NSWindow on the host, so SCStream can target it via SCContentFilter directly without going through simctl at all. swift helper as a daemon over UDS is the right call, just budget time for TCC, screen recording permission is sticky across user upgrades in weird ways and will bite you in support.

@MdSadiqMd