RFC: Adopt shared-texture mode for canvas sub-rect + 2D surround support

[dfattal]

RFC: Adopt shared-texture mode for canvas sub-rect + 2D surround

TL;DR

The runtime now supports _texture apps that publish a 3D canvas sub-rect of the window (xrSetSharedTextureOutputRectEXT) plus a full-window 2D surround texture that fills the non-canvas pixels (xrSetSharedTextureSurround2DEXT / xrSetSharedTextureSurround2DFenceEXT). End-to-end verified on Leia SR hardware in cube_texture_d3d11_win (KeyedMutex) and cube_texture_d3d12_win (fence).

Adopting it in Unity requires more than a C# wrapper — it's a pipeline-ownership shift. The plugin currently runs in handle mode (sharedTextureHandle = nullptr, runtime owns the editor HWND swap chain). The surround / canvas capability is only meaningful in shared-texture mode, where Unity allocates the multiview shared texture and owns presentation of the editor HWND. That's the design decision this RFC is about — the C# surface is the easy part once that's settled.

This issue is the prerequisite for closing #34 (canvas sub-rect) — they should ship together.

What landed in the runtime

• XR_EXT_win32_window_binding bumped to spec v7.
• §3.6 — xrSetSharedTextureSurround2DEXT(session, sharedTextureHandle, w, h) (D3D11, IDXGIKeyedMutex sync).
• §3.7 — xrSetSharedTextureSurround2DFenceEXT(session, texH, w, h, fenceH, awaitFenceValue) (D3D12, ID3D12Fence sync — keyed-mutex is E_NOINTERFACE on D3D12-native shared resources).
• D3D11 + D3D12 native compositors implement the strip blit in both shared-texture mode and HWND mode.

See XR_EXT_win32_window_binding.md, surround-2d-rollout.md, runtime PR #361.

The architectural question

Today the Unity plugin sets sharedTextureHandle = nullptr at session create — handle mode:

Mode	windowHandle	sharedTextureHandle	Who owns the HWND back buffer?	Surround possible?
Handle (current)	Editor HWND	NULL	Runtime — writes the weaved atlas directly into the HWND swap chain, presents	Yes (compositor copies strips into HWND back buffer), but the dst is runtime-owned so Unity-side composition / capture / post-process don't see it
Shared-texture (cube_texture)	Editor HWND	App-allocated shared NT-handle texture (panel-sized)	App — runtime writes weaved atlas + surround strips into the shared texture; app reads it, blits to its own back buffer, calls Present()	Yes (compositor copies strips into the shared texture); Unity gets full visibility

Surround technically works in both modes — but in handle mode the runtime is writing into a swap chain Unity doesn't own. Any Unity-side rendering Unity does to that HWND (gizmos, overlays, debug UI, the GameView's own chrome in the Editor) competes with the runtime's writes. Shared-texture mode resolves this by making Unity the presenter.

The decision is whether to switch Unity to shared-texture mode. The C# SetCanvasRect / SetSurround2D calls are mechanical once that's done.

Adoption paths

Path A — Big-bang switch (all Unity sessions)

Plugin always allocates a shared multiview texture, always passes it as sharedTextureHandle, always blits-and-presents Unity-side. Every Unity project that updates the plugin gets the new pipeline.

Pro: one code path. Surround / canvas just work everywhere.

Con: breaks the current pipeline for everyone in one release. Regressions in Editor display, GameView blit, DPI / multi-monitor handling, etc. surface for every project at once.

Path B — Opt-in (recommended)

Default stays handle mode. New project setting / OpenXR feature toggle: "Use shared-texture mode (enables canvas sub-rect + 2D surround)". When on, plugin allocates the shared texture, takes presentation, exposes canvas + surround APIs. When off, current behavior is preserved bit-for-bit.

Pro: zero forced migration. Projects opt in when they want the capability. Per-project setting maps cleanly to "this app uses a fixed-zone display."

Con: two code paths to maintain inside the plugin. Once shared-texture mode is shipped and stable, we can flip the default in a future major version.

The two paths diverge only in lifecycle / session-create wiring; the C# API + native plumbing for canvas + surround are identical.

Proposed C# API (once on shared-texture mode)

Component-based, attached to existing Unity Cameras — feels native to Unity devs and avoids a global manager singleton.

// Attach to the 3D-content Camera. Its viewport Rect (or the RectTransform
// of a UI element, if linked) becomes the canvas sub-rect — runtime weaves
// only into this region of the shared texture.
[AddComponentMenu("DisplayXR/3D Viewport")]
public sealed class DisplayXR3DViewport : MonoBehaviour {
    public enum CanvasSource { CameraViewport, RectTransform, Explicit }
    public CanvasSource source = CanvasSource.CameraViewport;
    public RectTransform rectTransform;
    public RectInt explicitPixelRect;
    // ...
}

// Attach to a 2D UI Camera (or assign a RenderTexture). Its output becomes
// the surround source. Plugin allocates a shared NT-handle RT matching the
// multiview shared texture's dims + format, hooks the camera's target to it,
// and registers via the spec-v6 KeyedMutex path (D3D11) or v7 fence path (D3D12)
// based on SystemInfo.graphicsDeviceType.
[AddComponentMenu("DisplayXR/2D Surround")]
public sealed class DisplayXRSurround : MonoBehaviour {
    public Camera surroundCamera;
    // Optional: explicit RT for advanced cases (UI Toolkit, Camera Stacking, etc.)
    public RenderTexture explicitTarget;
    // ...
}

For the static-API folks:

DisplayXR.SetCanvasRect(int x, int y, int w, int h);   // pixels in HWND client area
DisplayXR.SetSurround2D(RenderTexture rt);              // null clears

Both layers can ship together — the components are thin wrappers over the static API.

Open questions for the discussion

1. Path A vs Path B — opt-in or big-bang? Recommendation: B. (See above.)
2. D3D11 vs D3D12 selection — SystemInfo.graphicsDeviceType at plugin init, branch internally. The C# API surface is identical; only the native sync primitive differs.
3. Editor vs built player — issue Implement canvas sub-rect support for editor viewport weaving #34 calls out the editor's sub-rect-vs-window divergence. Same applies to surround: the editor's GameView occupies a sub-rect of the editor HWND, leaving real screen space for surround chrome (toolbars, scene panels). Built players usually have canvas == window, so surround is a no-op. Should the plugin auto-detect and skip surround setup in built players, or always enable?
4. Render pipelines — design needs to work across BRP / URP / HDRP. The component-based design above is RP-agnostic (uses standard Camera.targetTexture). Camera Stacking (URP) would be the natural way to layer surround on top of a base 3D camera.
5. Resize handling — Unity Editor windows are resized constantly. Surround texture dims must equal the shared multiview texture dims (compositor enforces equality). Reallocation strategy on WM_SIZE: tear down + recreate everything, or keep panel-worst-case dims and only update viewport math? cube_texture_d3d11_win does the latter; recommend the plugin do the same.
6. Texture format — runtime requires source format == dst format (no UNORM↔UNORM_SRGB cross-blit yet). For the BGRA-UNORM shared texture in cube_texture, surround must also be BGRA-UNORM. Unity's RenderTextureFormat.BGRA32 maps to this. Document.
7. Native handle export — Unity's Texture2D.GetNativeTexturePtr() returns ID3D11Texture2D* / ID3D12Resource*. Plugin queries IDXGIResource1::CreateSharedHandle (D3D11) or ID3D12Device::CreateSharedHandle (D3D12). For D3D12 the plugin must also create + share an ID3D12Fence per the v7 contract. Need to confirm Unity creates RTs with D3D11_RESOURCE_MISC_SHARED_NTHANDLE | _SHARED_KEYEDMUTEX / D3D12_HEAP_FLAG_SHARED flags — if not, plugin allocates natively and wraps via Texture2D.CreateExternalTexture.
8. Sync timing in D3D12 — per spec §3.7, app must Signal the fence on its queue after surround render submission and before the matching xrEndFrame. In Unity that means a CommandBuffer.IssuePluginEventAndData hook on the camera that records surround, with the native callback doing the Signal + the XR call. Workable but needs careful camera-event ordering.

Reference

Runtime spec	XR_EXT_win32_window_binding §3.6/§3.7
Runtime PR	#361 (merged)
D3D11 reference impl	cube_texture_d3d11_win — KeyedMutex path
D3D12 reference impl	cube_texture_d3d12_win — fence path
Rollout doc	surround-2d-rollout.md
Prereq issue	#34 (canvas sub-rect plumbing — closes when this lands)

Next steps

1. Discussion on Path A vs Path B.
2. Once agreed, file an implementation issue (or convert this one) with phases:
   • Phase 1: switch session create to shared-texture mode behind the feature flag, blit-and-present from C# side, no canvas/surround yet (pipeline-correctness milestone).
   • Phase 2: expose SetCanvasRect C# + native, closes Implement canvas sub-rect support for editor viewport weaving #34.
   • Phase 3: expose SetSurround2D C# + native (D3D11 KeyedMutex variant first).
   • Phase 4: D3D12 fence variant.
   • Phase 5: component wrappers (DisplayXR3DViewport, DisplayXRSurround).