Support physical iOS Target Devices with CoreMediaIO projection and WDA control

[drown0315]

Physical iOS Target Device Support PRD

Problem Statement

Ask UI currently treats a Target Device as an Android device or emulator. The Bridge verifies an Android target, starts scrcpy, streams H.264 into the Live App Surface, and forwards pointer and Android system controls back to the device. A Flutter developer working on a physical iPhone cannot use the same workbench loop: the iPhone screen is not available in the Device View, pointer input cannot reach the phone, iOS screenshots cannot be attached to Selection Comments, and the launch/session contracts reject the device before the workbench is usable.

Using Apple iPhone Mirroring for video and macOS CGEvent for input was considered, but it is not suitable for Ask UI's same-Mac workflow. Input requires iPhone Mirroring to become frontmost, which steals focus from the Ask UI browser and makes continuous browser gestures unreliable. Combining iPhone Mirroring video with WebDriverAgent input is also not a supported contract because an active XCUITest runner can interrupt iPhone Mirroring.

Ask UI needs production physical-iPhone support with a video path that does not depend on iPhone Mirroring and an input path that synthesizes touch on the device without moving the Mac cursor or changing browser focus.

Solution

Support a USB-connected physical iPhone as a Target Device by combining two independent, bridge-owned adapters.

The projection adapter uses a long-lived Swift helper to opt into CoreMediaIO iOS screen-capture devices, discover the matching AVCaptureDevice, capture frames with AVFoundation, encode them as low-latency H.264 with VideoToolbox, and send framed Annex B access units to the Dart Bridge through stdout. The Bridge parses the helper protocol and forwards raw H.264 payloads through the existing Device WebSocket. The Web workbench continues to decode and render the stream with WebCodecs.

The control adapter installs and runs a pinned Appium WebDriverAgent runner on the same iPhone. The Bridge sends coordinate gestures and supported system actions to WDA over a USB/CoreDevice-forwarded HTTP connection. WDA uses XCTest/XCUITest event synthesis on the device, so interaction does not require macOS Accessibility permission, does not consume the Mac cursor, and does not bring another Mac application frontmost.

The Bridge owns device identity resolution, capture and WDA lifecycle, gesture assembly, coordinate conversion, errors, cleanup, and session binding. The Web app owns Device View layout, pointer collection, WebCodecs decoding, platform-appropriate Surface Controls, and user-visible retry/setup states.

Before the feature enters the current product contract, a physical-device capability gate must prove that CoreMediaIO capture and WDA control remain active simultaneously on the same supported iPhone. Independent success of the video path and control path is insufficient evidence.

User Stories

1. As a Flutter developer, I want Ask UI to accept a physical iPhone reported by Flutter, so that I can inspect an iOS build in the same workbench used for Android.
2. As a Flutter developer, I want the Bridge Session to stay bound to one physical iPhone, so that video, Flutter Inspector data, screenshots, and input cannot silently target different devices.
3. As a Flutter developer, I want Ask UI to reject iOS Simulator as part of this physical-device capability, so that unsupported capture assumptions do not fail later.
4. As a Flutter developer, I want unsupported desktop and web Flutter targets to remain rejected, so that Target Device errors are immediate and specific.
5. As a Flutter developer, I want the target platform included in normalized Target Device metadata, so that the workbench presents the correct controls and diagnostics.
6. As a Flutter developer, I want Ask UI to detect when Xcode or required Apple device services are unavailable, so that I receive an actionable setup error.
7. As a Flutter developer, I want Ask UI to detect when the iPhone is not connected by USB, so that it does not wait indefinitely for video.
8. As a Flutter developer, I want Ask UI to detect when the iPhone has not trusted the Mac, so that I know which device action is required.
9. As a Flutter developer, I want Ask UI to detect when the iPhone must be unlocked, so that capture and WDA setup do not appear hung.
10. As a Flutter developer, I want Ask UI to map the Flutter/Xcode device identifier to the matching AVFoundation capture device, so that projection comes from the inspected phone.
11. As a Flutter developer, I want ambiguous capture-device name matches rejected, so that Ask UI never selects an arbitrary iPhone.
12. As a Flutter developer, I want device identity mismatches reported before control becomes ready, so that a click cannot reach a different phone from the displayed phone.
13. As a Flutter developer, I want the iPhone screen to appear in the Live App Surface, so that I can inspect the running Flutter app visually.
14. As a Flutter developer, I want projection to start automatically after the Bridge Session is ready, so that no separate video command is required.
15. As a Flutter developer, I want CoreMediaIO discovery and capture to occur in one long-lived helper process, so that a successful short-lived discovery cannot be invalidated by starting a different process.
16. As a Flutter developer, I want capture startup to wait for CoreMediaIO's asynchronous device publication with a bounded timeout, so that normal publication delay works without an infinite spinner.
17. As a Flutter developer, I want Camera permission failures distinguished from missing-device failures, so that I can fix the correct macOS setting.
18. As a Flutter developer, I want capture-device-busy errors to identify likely exclusive consumers, so that I can close QuickTime, OBS, or a stale helper.
19. As a Flutter developer, I want the capture helper to use the actual first-frame dimensions, so that the Device View matches each iPhone model.
20. As a Flutter developer, I want orientation and resolution changes reflected in complete metadata updates, so that video and input remain aligned.
21. As a Flutter developer, I want low-latency H.264 encoding, so that the Device View remains responsive while I interact with the phone.
22. As a Flutter developer, I want stale capture frames dropped rather than queued, so that latency does not grow during a long session.
23. As a Flutter developer, I want H.264 keyframes to include SPS and PPS, so that the browser can configure WebCodecs after startup or recovery.
24. As a Flutter developer, I want browser refresh to reuse the active capture process, so that CoreMediaIO does not repeatedly release and rediscover the phone.
25. As a Flutter developer, I want a new viewer to receive current metadata and a decodable keyframe, so that reconnect does not produce a permanent black screen.
26. As a Flutter developer, I want capture helper logs separated from binary stdout, so that diagnostics cannot corrupt the H.264 protocol.
27. As a Flutter developer, I want malformed or oversized helper frames rejected with a stable error, so that corrupted framing cannot cause unbounded Bridge memory use.
28. As a Flutter developer, I want the capture helper terminated when its owning Target Device session closes, so that the iPhone capture device is released.
29. As a Flutter developer, I want a bounded force-kill fallback during shutdown, so that a stuck AVCaptureSession cannot leave Ask UI unable to restart.
30. As a Flutter developer, I want Ask UI to prepare a pinned WebDriverAgent runner explicitly, so that control behavior does not change with an unreviewed upstream revision.
31. As a Flutter developer, I want Ask UI to use my selected Apple development team for WDA signing, so that the runner can be installed on my physical iPhone.
32. As a Flutter developer, I want missing Developer Mode, signing, provisioning, and trust failures reported separately, so that WDA setup is diagnosable.
33. As a Flutter developer, I want Ask UI to reuse an already installed compatible WDA runner, so that ordinary launches do not rebuild it.
34. As a Flutter developer, I want the Bridge to verify WDA health before enabling input, so that the UI does not accept gestures that cannot be delivered.
35. As a Flutter developer, I want the WDA session bound to the same device identifier used by Flutter launch, so that input reaches the displayed Target Device.
36. As a Flutter developer, I want a click in the Device View to produce a coordinate tap on the iPhone, so that Flutter buttons respond normally.
37. As a Flutter developer, I want pointer coordinates converted from video pixels into the current WDA window coordinate space, so that taps land accurately across resolutions and scale factors.
38. As a Flutter developer, I want letterboxed Device View areas ignored, so that clicks outside the rendered phone frame do not reach iOS.
39. As a Flutter developer, I want pointer down, move, up, and cancel collected into valid WDA gestures, so that the existing Web protocol can express iOS interaction.
40. As a Flutter developer, I want tap, long press, swipe, and drag duration preserved when a gesture is assembled, so that controls behave like direct touch.
41. As a Flutter developer, I want pointer move samples bounded and simplified before sending WDA Actions, so that long gestures do not create oversized HTTP requests.
42. As a Flutter developer, I want an interrupted gesture cancelled or released, so that WDA cannot leave a virtual finger pressed on the device.
43. As a Flutter developer, I want metadata changes to cancel an in-progress gesture, so that old coordinates are never applied in a new orientation.
44. As a Flutter developer, I want only one active pointer unless multi-touch is explicitly supported later, so that unsupported gesture combinations fail clearly.
45. As a Flutter developer, I want Home and other supported iOS navigation actions exposed as platform-appropriate Surface Controls, so that Android-only Back and Recents semantics are not mislabeled.
46. As a Flutter developer, I want unsupported iOS system actions hidden or disabled, so that every visible control has a verified WDA implementation.
47. As a Flutter developer, I want Select Widget mode to continue sending device gestures, so that Flutter Inspector can select the widget I touch.
48. As a Flutter developer, I want iOS interaction to leave the Ask UI browser focused, so that repeated workbench actions do not switch Mac applications.
49. As a Flutter developer, I want iOS interaction not to move the Mac cursor, so that Ask UI does not interfere with other desktop work.
50. As a Flutter developer, I want WDA action timeouts and device-busy failures returned as control errors without immediately destroying a healthy video stream, so that transient input failures remain recoverable.
51. As a Flutter developer, I want unrecoverable WDA disconnects reflected in controlReady, so that the Surface can disable input while video remains visible.
52. As a Flutter developer, I want an explicit retry to rebuild the WDA session without recreating the Flutter Bridge Session, so that control can recover independently.
53. As a Flutter developer, I want an explicit iOS screenshot captured for Selection Comments, so that visual context continues to accompany selected widgets.
54. As a Flutter developer, I want iOS snapshot failure to degrade to an unavailable attachment rather than block comment creation, so that the existing Selection Comment contract is preserved.
55. As a Flutter developer, I want device removal to fail projection and control with an actionable state, so that stale frames are not presented as a live phone.
56. As a Flutter developer, I want reconnect after USB removal to require an explicit retry, so that lifecycle ownership stays predictable.
57. As a Flutter developer, I want detailed capture and WDA diagnostics in Bridge logs, so that setup and runtime failures can be investigated without exposing raw tool output in the main UI.
58. As a Flutter developer, I want stable iOS error codes, so that the Web app and coding agent can explain the correct recovery step.
59. As an Ask UI maintainer, I want Android scrcpy behavior unchanged, so that adding iOS does not regress the existing Live App Surface.
60. As an Ask UI maintainer, I want platform adapters selected behind the existing DeviceStream boundary, so that WebSocket ownership does not depend on device-specific process details.
61. As an Ask UI maintainer, I want capture framing and WDA gesture assembly implemented as deep modules with small interfaces, so that protocol and lifecycle logic can be tested without hardware.
62. As an Ask UI maintainer, I want the Swift helper source and packaged binary versioned together, so that Bridge/helper compatibility is explicit.
63. As an Ask UI maintainer, I want release validation to reject a package missing the macOS capture helper, so that published iOS support is complete.
64. As an Ask UI maintainer, I want upstream WDA revision and license information recorded, so that dependency updates are deliberate and compliant.
65. As an Ask UI maintainer, I want physical-device acceptance evidence recorded by model, iOS version, Xcode version, capture ID, and WDA revision, so that the support claim is reproducible.
66. As an Ask UI maintainer, I want a combined capture-and-control gate, so that independent mocks cannot falsely prove physical-iPhone support.

Implementation Decisions

• Use the established domain language: Target Device, Bridge Session, Live App Surface, Device View, and Surface Controls.
• Generalize Target Device from Android-only to an explicit platform classification that currently supports Android and physical iOS.
• Keep the startup contract based on vmServiceUri, projectRoot, and deviceId; do not add a second iOS-only device selector to the Web URL.
• Treat one Bridge Session as bound to exactly one Target Device and reject platform or identifier changes for the same Flutter app session.
• Continue using Flutter's machine-readable device listing as the initial source for development device identity and display name.
• Accept physical iOS target platforms and continue rejecting iOS Simulator in this PRD.
• Keep Android availability, scrcpy lifecycle, and control behavior in the Android adapter.
• Add a platform-selecting DeviceStream factory rather than adding iOS branches throughout the scrcpy implementation.
• Build the physical-iOS implementation from two independently testable modules: a capture session and a WDA control session.
• Use the CoreMediaIO/AVFoundation/VideoToolbox projection design documented in the accepted iOS screen-projection technical guide.
• Do not use Apple iPhone Mirroring, ScreenCaptureKit window capture, macOS CGEvent input, AirPlay, ReplayKit, or ask_ui_runtime pointer synthesis for this capability.
• Implement the capture layer as a Swift helper because CoreMediaIO, AVFoundation, and VideoToolbox are native macOS framework boundaries.
• Opt into kCMIOHardwarePropertyAllowScreenCaptureDevices and perform discovery and streaming in the same long-lived helper process.
• Use xcrun xctrace list devices only for USB development identity and name mapping; do not treat xctrace output as evidence that AVFoundation has published a capture device.
• Match an exact AVFoundation capture identifier first and an exact unique device name second; reject ambiguous matches.
• Require USB connection, trust, unlocked/awake device state, and Camera permission for the projection path.
• Use one active capture owner for a physical iPhone. A Device WebSocket viewer attaches to that owner and does not directly own the native capture process.
• Keep capture alive across viewer refresh/reconnect while its owning Bridge Target Device session remains alive.
• Configure AVFoundation to discard late frames and never build an unbounded capture queue.
• Encode H.264 with VideoToolbox in real-time mode, disabled frame reordering, constrained baseline profile, configurable frame rate/bitrate, and periodic keyframes.
• Convert VideoToolbox AVCC output to Annex B and prepend SPS/PPS to every emitted IDR access unit.
• Send ready metadata only after the first frame establishes real dimensions.
• Use the technical guide's stdout contract: one JSON metadata line followed by length-bounded binary access-unit envelopes carrying flags and microsecond timestamps.
• Reserve stdout exclusively for the capture protocol and stderr exclusively for diagnostics.
• Parse helper output incrementally; process pipe chunks are never assumed to align with protocol records.
• Enforce a bounded metadata line, bounded parser buffer, and 16 MiB maximum access unit.
• Strip the helper envelope in the Bridge and forward the Annex B access unit as the existing Device WebSocket binary payload, preserving the current WebCodecs pipeline.
• Emit complete metadata updates when dimensions or orientation change, recreate the encoder, and force a new configuration-bearing keyframe.
• Keep WebCodecs as the browser decode path and retain low-latency delta-frame dropping under decoder pressure.
• Pin a reviewed Appium WebDriverAgent revision and expose its version in diagnostics, not in high-frequency UI.
• Preserve upstream WDA license and attribution in distributed artifacts or setup output.
• Use Xcode signing and provisioning for the user-selected physical iPhone; Ask UI does not install Xcode, create an Apple account, or bypass Developer Mode.
• Make WDA preparation explicit and reusable. Ordinary Ask UI launch should health-check and reuse a compatible installed runner rather than rebuild on every session.
• Use the Flutter/Xcode development UDID as the WDA destination and require it to resolve to the same name selected by the capture helper.
• Reach WDA through a local USB/CoreDevice relay and keep its HTTP endpoint private to the local Bridge.
• Create and maintain one WDA automation session per active physical-iOS Target Device control session.
• Use W3C Actions or equivalent WDA coordinate endpoints that synthesize events through XCTest/XCUITest on the device.
• Convert Device View coordinates through normalized screen position into WDA's current window size instead of assuming capture pixels equal XCTest points.
• Assemble the existing pointer down/move/up/cancel stream into a bounded complete gesture before submitting it to WDA.
• Preserve gesture timestamps, simplify excessive move samples, and impose maximum duration and point-count limits.
• Cancel incomplete gestures on WebSocket close, metadata change, control failure, or explicit browser cancel.
• Keep single-pointer interaction as the current contract. Multi-touch must not be inferred from WDA capability alone.
• Add platform and control-capability fields to Device metadata so the Web app renders verified Surface Controls rather than Android-specific Back/Home/Recents for every device.
• Keep a healthy video stream alive across recoverable WDA control errors; update control readiness independently when possible.
• Fail the complete physical-iOS Device session when the phone disconnects or capture becomes unusable.
• Use WDA's on-device screenshot endpoint for explicit physical-iOS Selection Comment snapshots, preserving the existing unavailable fallback behavior.
• Use stable error families for dependency/setup, permission, device identity, capture startup/runtime, video encoding, WDA signing/start/session, and control delivery.
• Do not expose raw signing output, provisioning secrets, environment variables, or complete subprocess command lines through Web error messages.
• Package a compatible macOS capture helper with the Bridge release and validate its presence and executable architecture before publishing.
• Manage helper SIGTERM/SIGINT shutdown, pending-start cancellation, bounded wait, and SIGKILL fallback inside the Bridge-owned capture launcher.
• Preserve the existing one-active-Device-WebSocket rule for a Bridge Session.
• Require a real physical-device gate before changing documentation or product language to claim physical-iOS support.
• The combined gate must prove sustained CoreMediaIO H.264 capture while WDA starts, creates a session, and performs tap, long press, swipe, drag, and Home actions on the same iPhone.
• If simultaneous WDA and CoreMediaIO operation fails on a supported target, mark the combined capability unsupported and revise the architecture; do not silently fall back to iPhone Mirroring or CGEvent.

Testing Decisions

• Automated tests should verify externally observable behavior at module boundaries: normalized device decisions, framed bytes, process lifecycle calls, HTTP requests, Device WebSocket messages, coordinate transforms, UI capability states, snapshot results, and stable errors. Tests should not assert private helper class names or incidental internal state.
• The platform-aware Target Device checker should be tested with Flutter machine-output fixtures covering Android, physical iOS, iOS Simulator, unsupported targets, exact identifier matching, missing names, malformed output, and command failure.
• The platform DeviceStream factory should be tested to select scrcpy for Android, CoreMediaIO/WDA for physical iOS, and reject unsupported platforms without starting native processes.
• The capture identity resolver should be tested with xctrace fixtures covering exact UDID, exact unique name, duplicate names, offline devices, and a direct AVFoundation capture identifier.
• The helper protocol parser should be tested with metadata split across chunks, metadata and first frame in one chunk, every possible header split, payload splits, multiple frames in one chunk, invalid UTF-8/JSON, oversized metadata, oversized payloads, early EOF, and stderr-specific error precedence.
• The capture launcher should use a fake process boundary to test one-process discovery/streaming, startup timeout, pending-start cancellation, viewer reconnect without process restart, session close, graceful termination, forced termination, and late process completion.
• AVCC-to-Annex-B conversion, SPS/PPS extraction, IDR flags, timestamp encoding, encoder recreation, and shutdown should have Swift unit tests independent of a physical device.
• WDA setup tests should verify pinned revision handling, reusable installation detection, device/team command construction, Developer Mode/signing/provisioning error normalization, relay lifecycle, and cleanup without invoking a real Apple account.
• The WDA HTTP client should be tested against a fake local server for health, session creation, window-size lookup, actions, Home, screenshots, timeouts, malformed responses, session invalidation, and retry boundaries.
• Gesture assembler tests should cover tap, long press, swipe, drag, cancel, missing down, duplicate down, metadata change, excessive move samples, maximum duration, coordinate bounds, and orientation-aware mapping.
• Control lifecycle tests should verify that recoverable WDA errors disable or fail control without corrupting a healthy capture stream, while device removal fails the complete Device session.
• Device WebSocket tests should cover iOS ready metadata, platform capabilities, binary H.264 forwarding without helper envelope bytes, metadata updates, control errors, retry, single active connection, and cleanup.
• Snapshot tests should verify WDA PNG decoding, session-scoped file creation, size handling, unavailable fallback, and unchanged Android ADB screenshot behavior.
• Web protocol tests should cover platform/capability metadata, iOS Surface Control selection, disabled control states, coordinate mapping, gesture cancellation, and unchanged Android controls.
• Existing scrcpy stream tests, Device WebSocket tests, snapshot tests, WebCodecs parser tests, Device View geometry tests, and Surface Control tests are the closest prior art and should remain green.
• A physical-device acceptance test is mandatory and cannot be replaced by mocks. It must record iPhone model, iOS version, macOS version, Xcode version, Flutter device id, AVFoundation capture id, WDA revision, first-IDR latency, sustained frame rate, action latency, and cleanup behavior.
• The physical gate must start capture first, then start WDA, and verify that frame timestamps and byte counts continue increasing while each action visibly changes the captured screen.
• Physical acceptance must also cover browser refresh, orientation change, WDA restart, WDA stop while capture continues, USB removal, unlock/lock behavior, Camera permission denial, capture-device contention, and Bridge shutdown with no lingering helper or relay.
• Before completion, run all Dart tests, all Web tests, Web lint/typecheck/build, Swift helper tests/build, and the documented physical-device acceptance sequence.

Out of Scope

• iOS Simulator support.
• Wireless-only iPhone projection or control.
• Multiple simultaneous physical iPhones.
• Multiple simultaneous control owners or multi-user sessions.
• iPhone audio capture.
• Internet streaming, authentication, TURN, or WebRTC.
• Apple iPhone Mirroring, AirPlay, ScreenCaptureKit window capture, or macOS CGEvent injection.
• ReplayKit integration into the inspected Flutter app.
• ask_ui_runtime pointer-event injection.
• Using WDA MJPEG or screenshot polling as the primary Live App Surface video path.
• WDA Accessibility tree as a replacement for the Flutter Widget Tree.
• Label-based WDA element selection in the Ask UI product UI.
• Multi-touch and pinch gestures.
• General-purpose automation of other iOS apps.
• Arbitrary keyboard/text forwarding beyond the existing Live App Surface control contract.
• Automatic installation of Xcode, Flutter, Apple certificates, Apple accounts, or Developer Mode.
• Bypassing Apple code-signing, provisioning, trust, privacy, or device-security requirements.
• Silent fallback to a different phone, iPhone Mirroring, CGEvent, or an unpinned WDA revision.
• Claiming support for a macOS/iOS/Xcode combination that has not passed the combined physical-device gate.

Further Notes

• The accepted projection reference is docs/ios-screen-projection-technical-guide.md on the feat/ios-screen-projection branch. Its verified capture data establishes CoreMediaIO/AVFoundation/VideoToolbox projection independently, but it does not establish simultaneous WDA operation.
• Appium WebDriverAgent source confirms that W3C pointer actions are converted to XCUICoordinate, XCPointerEventPath, and XCSynthesizedEventRecord and delivered through XCTest. This supports true physical-device input without Mac focus changes.
• The current evidence status is: CoreMediaIO projection supported independently, WDA physical-device control supported independently, simultaneous operation unknown until the Ask UI physical gate runs.
• The architecture intentionally keeps video and control behind separate deep modules so either external boundary can be replaced without changing the Device WebSocket or WebCodecs renderer.
• Android remains a first-class platform adapter and must not be described as a fallback for iOS.