Windows: `rdc capture` silently times out when a system RenderDoc is also installed (duplicate implicit layer) + unsubstituted `@VULKAN_*@` vars in generated renderdoc.json

[borisbat]

What happened?

On Windows with a system RenderDoc install present, rdc capture injects and connects but never produces a capture — it just prints waiting for capture... (5s/10s/…) until timeout waiting for capture, even though RenderDoc's frame-counter overlay is visible on the target window (so RenderDoc is hooking present).

Steps to reproduce

1. Have RenderDoc installed system-wide (this registers an implicit layer VK_LAYER_RENDERDOC_Capture under HKLM\SOFTWARE\Khronos\Vulkan\ImplicitLayers).
2. uv tool install rdc-cli then rdc setup-renderdoc (builds rdc's own renderdoc and registers its own VK_LAYER_RENDERDOC_Capture implicit layer under HKCU).
3. rdc capture -o out.rdc -- <any windowed Vulkan app> (also tried --frame N, --trigger, and the no-frame TriggerCapture path).

Expected: a .rdc is written.
Actual: timeout, no NewCapture message. Overlay shows on the window.

Root cause (two compounding problems)

(a) setup-renderdoc installs a renderdoc.json with unsubstituted CMake template variables.
The generated manifest at %LOCALAPPDATA%\rdc\renderdoc\renderdoc.json contains literal placeholders:

• enable_environment: { "@VULKAN_ENABLE_VAR@": "1" } — in both the default (v1.41) and --version v1.44 builds.
• name: "@VULKAN_LAYER_NAME@" and disable_environment: "DISABLE_VULKAN_RENDERDOC_CAPTURE_@RENDERDOC_VERSION_MAJOR@_@RENDERDOC_VERSION_MINOR@" — additionally broken in the v1.44 build (the default v1.41 build does substitute name correctly, but not enable_environment).

Because no real env var is named @VULKAN_ENABLE_VAR@, rdc's own implicit layer can never self-enable, and in v1.44 its layer name isn't even VK_LAYER_RENDERDOC_Capture. So rdc's layer is effectively dead and capture leans entirely on ExecuteAndInject.

(b) Duplicate-layer split-brain.
With a system RenderDoc also installed, two implicit layers named VK_LAYER_RENDERDOC_Capture are registered (system HKLM + rdc HKCU), pointing to different renderdoc.dll files. The Vulkan loader name-dedups and loads the system DLL to hook the app's Vulkan calls (hence the visible overlay), while ExecuteAndInject + CreateTargetControl go to rdc's own-built DLL. The instance that hooks present ≠ the instance rdc controls, so QueueCapture/TriggerCapture never reach the hooking layer.

Confirmed with a minimal ExecuteAndInject + CreateTargetControl probe: the connection succeeds (valid ident + PID) but receives zero RegisterAPI / CapturableWindowCount messages — rdc's controlled instance never sees a graphics API, because the system instance is the one bound to Vulkan. (qrenderdoc captures the same app fine, because it injects the same DLL its implicit layer loads.)

Recommended fix

1. Fix manifest generation — run the CMake configure_file step (or post-process renderdoc.json after build) so @VULKAN_LAYER_NAME@, @VULKAN_ENABLE_VAR@, @RENDERDOC_VERSION_MAJOR@/MINOR@ are substituted. The --version v1.44 path regresses even the layer name, so non-default versions appear to skip/break the configure step.
2. Detect the conflict in rdc doctor — scan HKLM+HKCU ImplicitLayers (and the Linux/macOS manifest dirs) for more than one VK_LAYER_RENDERDOC_Capture and warn, listing each manifest path + version. Two installs make injected capture unreliable, and today doctor is fully green while capture is broken.
3. Make rdc capture force its own layer for the spawned child — set VK_ADD_LAYER_PATH + VK_LOADER_LAYERS_ENABLE=VK_LAYER_RENDERDOC_Capture pointed at rdc's manifest dir and VK_LOADER_LAYERS_DISABLE the foreign one for the child process, so the hooking layer == the injected/controlled DLL. At minimum, fail fast with an actionable message when a foreign RenderDoc layer is detected, instead of timing out silently.

Workaround (worth documenting): capture via the system RenderDoc (F12 / qrenderdoc, after set ENABLE_VULKAN_RENDERDOC_CAPTURE=1) and use rdc open to inspect — replay is unaffected since it uses the python module, not the implicit layer.

rdc doctor output

[ok] python: 3.12.13
[ok] platform: windows
[ok] renderdoc-module: version=1.44
[ok] replay-support: renderdoc replay API surface found
[ok] renderdoccmd: C:\Program Files\RenderDoc\renderdoccmd.exe (renderdoccmd x64 v1.44 ...)
[ok] win-python-version: MSBuild renderdoc.pyd found ...
[ok] win-vs-build-tools: Visual Studio Build Tools found ...
[ok] win-renderdoc-install: RenderDoc found at C:\Program Files\RenderDoc\renderdoc.dll
[ok] win-vulkan-layer: registered at C:\Users\...\AppData\Local\rdc\renderdoc\renderdoc.json
[ok] renderdoc-variant: upstream (1.44)

(All green — doctor does not notice the second, system-wide RenderDoc layer, which is the crux of the bug.)

OS / Platform

Windows 11 (system RenderDoc v1.44 installed alongside rdc's built module)

Python version

3.12.13 (rdc's bundled venv)

rdc-cli version

0.5.6

Capture file

N/A — the failure is at capture time; no .rdc is produced. Reproduces with any windowed Vulkan app; in my case a volk-based daslang/Vulkan renderer.