macOS support for DisplayXRTransparentOverlay (visual transparency; click-through deferred)

[dfattal]

Summary

Add macOS support to DisplayXRTransparentOverlay so the chroma-key transparent overlay mode (currently Windows-only, shipped via #57) works on Mac. This issue covers visual transparency only — click-through is described below as a follow-on phase with its own platform-specific obstacle.

Background — runtime side is already done

The DisplayXR runtime shipped macOS-side transparency support in v1.3.0 (DisplayXR/displayxr-runtime PR #4 train):

• XR_EXT_cocoa_window_binding v5 added transparentBackgroundEnabled field (macOS analog of the Win32 binding's same field).
• comp_metal_compositor_create plumbs the bit through and configures both app-owned views (CAMetalLayer.opaque=NO) and runtime-owned NSWindows (NSWindow.opaque=NO + clearColor background + CAMetalLayer.opaque=NO).
• sim_display_processor_metal.m declares is_alpha_native = true. SBS / anaglyph / blend shaders write atlas alpha straight through, so no chroma-key fill/strip is needed on Mac — alpha=0 pixels from the app reach the desktop directly via Cocoa per-pixel transparency. Chroma-key is now Leia-DP-internal (Windows-only path).
• End-to-end validated by the cube_handle_metal_macos test app under DISPLAYXR_TRANSPARENT_BG=1.

So on Mac the plugin just needs to (a) flip the new extension bit, and (b) configure the Unity-owned NSWindow correctly. The runtime only handles CAMetalLayer.opaque for app-provided views — the NSWindow itself is the app's responsibility.

Plugin-side architecture (visual transparency)

Shared C# API (no platform branching)

DisplayXRTransparentOverlay.cs keeps its current public surface:

• enabled, chromaKeyColor, etc.
• On macOS, chromaKeyColor is a no-op (sim_display is alpha-native). Keep the property for API symmetry; document that it's ignored on Mac.

Native split

native~/displayxr_transparent_overlay.cpp      # shared dispatcher
native~/displayxr_transparent_overlay_win32.cpp  # existing Windows path
native~/displayxr_transparent_overlay_macos.mm   # NEW

The shared dispatcher just routes to the platform implementation by #if defined(_WIN32) / #elif defined(__APPLE__). Existing Windows code unchanged.

macOS native implementation outline

// displayxr_transparent_overlay_macos.mm

// 1. Session creation — set the cocoa_window_binding v5 bit BEFORE xrCreateSession.
//    Same lifecycle hook as the Win32 path (RequestTransparentSession from
//    [RuntimeInitializeOnLoadMethod(SubsystemRegistration)]).
void displayxr_macos_request_transparent_session(bool enabled) {
    s_transparent_requested = enabled;
}

// In the xrCreateSession hook (already exists in native plugin):
//   - On Win32 path: append XrWin32WindowBindingCreateInfoEXT with transparentBackgroundEnabled
//   - On macOS path: append XrCocoaWindowBindingCreateInfoEXT v5 with transparentBackgroundEnabled
// The append-extension-struct pattern is identical; only the struct type differs.

// 2. NSWindow configuration — runtime sets CAMetalLayer.opaque on the layer
//    Unity provides, but it does NOT touch the NSWindow itself for app-owned
//    views. Plugin must flip:
//      [unityWindow setOpaque:NO];
//      unityWindow.backgroundColor = [NSColor clearColor];
//    Timing: do this AFTER Unity creates its window but BEFORE the first
//    frame submission. Earliest reliable hook is the OpenXR session_begin
//    callback (or a post-OnEnable runtime callback).
void displayxr_macos_configure_unity_nswindow(void) {
    NSWindow *w = displayxr_get_unity_nswindow();  // via NSApp.windows enumeration
    if (!w) return;
    [w setOpaque:NO];
    w.backgroundColor = [NSColor clearColor];
}

C# wiring

DisplayXRTransparentOverlay.OnEnable (on macOS branch):

1. Calls displayxr_macos_request_transparent_session(true) from [RuntimeInitializeOnLoadMethod(SubsystemRegistration)] — runs before xrCreateSession, plugin appends XrCocoaWindowBindingCreateInfoEXT with the bit set.
2. After the session is up (session_begin), plugin calls displayxr_macos_configure_unity_nswindow().
3. On Camera.preCull (or similar early hook), the C# layer sets the active camera's clearFlags = Color and backgroundColor = (0,0,0,0). Same pattern as the Win32 path.

Risks / open questions

• Finding Unity's NSWindow reliably. Unity creates one NSWindow per Game View / build target. NSApp.windows enumeration plus a filter on title/class should work; verify against both Editor and built-app cases.
• Editor preview vs Play mode. The standalone preview window owns its own NSWindow (runtime-side, configured by the runtime). The transparent overlay scenario applies only to built apps and Play mode in the Editor — preview window already gets correct treatment because the runtime manages its window. Verify the preview path doesn't regress when this flag is set.
• Camera-clear color in URP/HDRP. BiRP Camera.clearFlags = SolidColor + backgroundColor is well-understood; URP/HDRP may need a custom render pass for true alpha=0 background. The current test repo (displayxr-unity-test-transparent) is BiRP only — Mac parity ships there first.

Verification flow

After building and running on Mac with the feature enabled:

1. Cube (or tiger) renders in stereo with alpha=0 in unrendered regions.
2. Desktop apps behind Unity's window are visible through the transparent regions.
3. Clicks on Unity content still hit Unity (click-through deferred — see below).
4. The cube_handle_metal_macos test app's behavior is the proof-of-concept for the rendering path; Unity should match it visually.

Out of scope for this issue — click-through (follow-on phase)

Per-pixel click-forwarding to apps underneath is fundamentally harder on macOS than on Windows. Windows gives you HTTRANSPARENT as a free OS primitive: WM_NCHITTEST returns it for chroma-key regions, the OS automatically delivers the click to whatever app is below. ~5 lines of code, no permissions.

macOS has no equivalent:

• NSWindow.ignoresMouseEvents = YES is binary (whole window), not per-pixel.
• Custom NSView.hitTest: returning nil for transparent pixels only prevents your app from receiving the event — it does NOT forward to the app underneath.
• To actually forward the click, you must synthesize a CGEventCreateMouseEvent and post it via CGEventPost(kCGHIDEventTap, ...). This requires the user to grant Accessibility permission (same prompt that screen-recorders and keyboard-automation tools use). Right-drag-to-move-window needs its own NSWindow-based path.

So follow-on phase will be tracked as a separate issue once visual transparency lands. Two paths to consider then:

1. Visual-only forever — Unity catches all clicks in the chroma-key region. Acceptable if the desktop-avatar use case can tolerate the user clicking through "around" the avatar to reach apps.
2. CGEventPost + Accessibility prompt — full parity with Win32 behavior, at the cost of a system permission prompt on first run.

Cross-references

• Plugin Win32 implementation: Runtime/DisplayXRTransparentOverlay.cs + native~/displayxr_transparent_overlay_win32.cpp (current naming may differ).
• Runtime macOS work: DisplayXR/displayxr-runtime PR Test camera-centric mode end-to-end with sim_display #4 train (commits dac135acf, d2d83ef1b, 6adcc065d).
• Original Win32 plugin issue: Add transparent overlay mode for desktop avatar use case #57 (closed, shipped).
• Original Win32 runtime issue: DisplayXR/displayxr-runtime#191 (closed, shipped).