Editor Preview blank on Vulkan editor device — implement Vulkan standalone shared-texture backend (#122 follow-up)

[dfattal]

Summary

The standalone editor Preview window (Window > DisplayXR > Preview) renders blank when the Unity editor is running on the Vulkan graphics API. It works on D3D11/D3D12. Built apps on Vulkan are unaffected (PR #122/#123 handles the hooked/built-app path).

Root cause

The standalone session does not use the editor's graphics device — on Windows it hardcodes its own D3D12 device (create_standalone_d3d12_backend(), native~/displayxr_standalone.cpp:1016) and renders the runtime output into a D3D12 atlas. It then shares that atlas into the editor's device for zero-copy display via a cross-API shared handle.

That hand-off only bridges D3D12 ↔ D3D11:

• DisplayXRPreviewSession.cs:259 passes the editor's RenderTexture.GetNativeTexturePtr() to displayxr_standalone_set_unity_device(...).
• On a Vulkan editor, that pointer is a Vulkan handle, so the D3D12 backend's set_unity_device (native~/displayxr_standalone_d3d12.cpp:510) fails its ID3D12Resource QueryInterface → unity_device stays null (no crash — guarded since UPM cannot add package via git URL on Windows and macOS #35).
• create_atlas_bridge then early-returns (if ((!unity_device && !unity_d3d11_device) || !d3d12_device) return;, displayxr_standalone_d3d12.cpp:236) → displayxr_standalone_get_atlas_bridge_texture returns null → CreateAtlasBridge (DisplayXRPreviewSession.cs:562) gets bridgePtr == IntPtr.Zero → no preview texture → blank window.

So the SA session itself runs fine on a Vulkan editor; only the D3D12→Vulkan zero-copy display hand-off fails. The Vulkan standalone backend (native~/displayxr_standalone_vulkan.cpp) is a pure stub (create_device/create_shared_texture → false/nullptr) — marked Phase 2 in docs~/roadmap/full-gfx-backends-refactor-plan.md.

Fix options

• (A) Implement the Vulkan standalone shared-texture backend (displayxr_standalone_vulkan.cpp). On Windows this means either importing the D3D12-shared NT handle as a VkImage via VK_KHR_external_memory_win32 (vkGetMemoryWin32HandleKHR/import), or rendering the SA session through a Vulkan device and exporting a Win32 handle the editor imports. Roughly Phase-2 scope (~hundreds of lines + Vulkan SDK + runtime advertising Vulkan display support). This is the proper fix.
• (B) CPU-readback display path for the preview — API-independent (works on any editor device) but adds a GPU→CPU→GPU round-trip per frame.
• (C) Guard + message in the Preview window (detect editor graphicsDeviceType == Vulkan, show an actionable message instead of a silent blank). Considered and declined for now.

Workaround

Relaunch the Unity editor with -force-d3d12 (the editor's API is independent of the build target — your Vulkan player build is unaffected). Built Vulkan apps run correctly.

Files

• Editor/DisplayXRPreviewSession.cs — set_unity_device call (259), CreateAtlasBridge (562)
• native~/displayxr_standalone.cpp — backend selection (1016), set_unity_device export (1876)
• native~/displayxr_standalone_d3d12.cpp — set_unity_device (510), create_atlas_bridge early-return (236)
• native~/displayxr_standalone_vulkan.cpp — stub backend (Phase 2)

Related: #122 / PR #123 (Windows Vulkan hooked-path backend — built apps only).