From 621389f72867c34082b9df39b1727dd033a36c46 Mon Sep 17 00:00:00 2001 From: Alexey Lysenko Date: Sun, 6 Sep 2026 20:51:37 +0300 Subject: [PATCH] wrapper: emulate VK_EXT_host_query_reset when the base driver lacks it DXVK's D3D11 path forces this feature on and cannot be talked out of it. D3D11Device::GetDeviceFeatures sets enabled.extHostQueryReset.hostQueryReset = VK_TRUE; inside the >= D3D_FEATURE_LEVEL_9_1 block, and DxvkAdapter::checkFeatureSupport then gates device creation on (supported || !required). So on a driver that does not advertise it, every feature level is refused, not only the high ones -- which is why on Tegra X1 under the NVIDIA proprietary Android driver D3D9 ran while every D3D10/D3D11 title died at device creation with E_INVALIDARG, Unity reporting "d3d11: failed to create device and context (80070057)". This is the only capability that behaves that way. Everything else DXVK asks for in that function, memoryPriority and timelineSemaphore included, is assigned as `enabled.x = supported.x`, so it is required only when it is already present and can never fail the check. Read from the version the device actually runs, DXVK-Sarek v1.11.0, and confirmed on hardware: with the emulation on and nothing else, D3D10 and D3D11 pass in both bitnesses. leegao's bionic-vulkan-wrapper reached the same conclusion independently and is the precedent for the approach. It advertises exactly this extension, under a comment naming DXVK as the reason: // Needed by dxvk exts->EXT_host_query_reset = true; and masks the feature back off in the chain forwarded to the driver, which is what process_pnext_chain does here. It advertises nothing for memory priority. Where this patch differs is the part behind the advertisement. That wrapper implements no vkResetQueryPool, which costs nothing on its targets -- Adreno 6xx and Mali G52 onwards support the feature natively, so the advertisement is a no-op there and the driver's own entry point is used. On a driver that genuinely lacks it there is nothing behind the claim, and DXVK calls the entry point on a hot path: DxvkGpuQueryManager::writeTimestamp and beginSingleQuery reset a query before every timestamp and every query begin. So it is emulated rather than merely claimed. vkResetQueryPool must have taken effect by the time it returns -- it "sets the status of query indices [firstQuery, firstQuery + queryCount - 1] to unavailable" -- so it records vkCmdResetQueryPool into a command buffer, submits it, and waits on a fence. Pool, command buffer and fence are created once at device creation. https://docs.vulkan.org/refpages/latest/refpages/source/vkResetQueryPool.html The emulation inherits the entry point's own valid usage, which does most of the work: the application must already have completed submitted commands referring to the range, and must not have that range in use from another thread. So the reset never races the application's use of those queries. What it can race is unrelated work on the queue it submits to, and that is the whole design problem. Sharing the application's queue couples the reset to whatever is already queued on it: a legal application submit can wait on a host event the application only signals after its reset returns, and the reset then waits for work that is waiting for the reset. A probe arms a vkCmdWaitEvents on the application queue and releases the event from a watchdog thread after one second, so this shows up as latency rather than a hang. On Tegra a shared-queue implementation waited 1001 ms in five runs out of five -- the full watchdog, every time. So the emulation asks vkCreateDevice for one extra queue in a family the application already requested, and submits there. The application never learns about it: wrapper_create_device_queue builds wrapper_queue objects from the application's own VkDeviceCreateInfo, not from the one forwarded to the driver. Same probe, same device: 0.2 ms instead of 1001 ms. If the family has no spare queue the emulation falls back to sharing the application's first graphics/compute queue and restores the serialisation that needs, because a VkQueue is externally synchronised and the application cannot synchronise against a submit it does not know about. Refusing to create the device there would turn a slow device into no device at all. wrapper_query_reset_owns_queue is false whenever the queue is private, so the private path pays nothing for the fallback. vkDeviceWaitIdle is serialised against the internal queue for the same reason: its host synchronisation requirement covers "all VkQueue objects created from device", and one of them is invisible to the application. https://docs.vulkan.org/refpages/latest/refpages/source/vkDeviceWaitIdle.html Only the core entry point is defined. The generated table lists vkResetQueryPool and vkResetQueryPoolEXT separately but both resolve to one dispatch slot, and defining the alias as well trips the "disp[disp_index] == NULL" assert in vk_device_dispatch_table_from_entrypoints and kills device creation. That mistake was made and caught on the device. Gated like the native BCn preference and the AHB ownership release: on the driver, with WRAPPER_EMULATE_HOST_QUERY_RESET as a tri-state override, and never applied over native support. The emulation is not free -- every host reset is a command buffer and a fence wait, about 80-120 us on this hardware -- so the kill switch exists for a client that does not need it. --- src/vulkan/wrapper/wrapper_device.c | 271 +++++++++++++++++++ src/vulkan/wrapper/wrapper_physical_device.c | 19 ++ src/vulkan/wrapper/wrapper_private.h | 10 + 3 files changed, 300 insertions(+) diff --git a/src/vulkan/wrapper/wrapper_device.c b/src/vulkan/wrapper/wrapper_device.c index b99530a7..9b6a46db 100644 --- a/src/vulkan/wrapper/wrapper_device.c +++ b/src/vulkan/wrapper/wrapper_device.c @@ -178,6 +178,12 @@ static void process_pnext_chain(VkBaseInStructure *create_info, struct wrapper_p WRAPPER_LOG(info, "Unlinking VkPhysicalDeviceRobustness2FeaturesEXT from pNext chain"); unlink_vk_struct(create_info, ¤t, &prev); continue; + case VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_HOST_QUERY_RESET_FEATURES: + if (pdevice->base_supported_features.hostQueryReset) + break; + WRAPPER_LOG(info, "Unlinking VkPhysicalDeviceHostQueryResetFeatures from pNext chain"); + unlink_vk_struct(create_info, ¤t, &prev); + continue; case VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_DYNAMIC_RENDERING_UNUSED_ATTACHMENTS_FEATURES_EXT: if (pdevice->base_supported_extensions.EXT_dynamic_rendering_unused_attachments) break; @@ -231,6 +237,226 @@ static void process_pnext_chain(VkBaseInStructure *create_info, struct wrapper_p } } +/* True when the emulation shares this queue with the application, so submits on + * it must be serialised against the internal one. A private queue is invisible + * to the application, so no wrapper_queue ever carries its handle and this + * predicate stays false -- the shared-queue cost is paid only in the fallback. */ +static bool +wrapper_query_reset_owns_queue(const struct wrapper_queue *queue) +{ + const struct wrapper_device *device = queue->device; + return device->query_reset_queue != VK_NULL_HANDLE && + device->query_reset_shared_queue && + device->query_reset_queue == queue->dispatch_handle; +} + +/* Reserve a queue for the reset submits. Preferred: one extra queue in a + * family the application already asked for, which the application never sees, + * so a reset cannot wait behind application work. If the family is exhausted + * -- the application asked for every queue it has -- fall back to sharing the + * application's first queue, which is what this emulation did before and is + * measurably correct but latency-coupled. Returning an error here instead + * would turn a slow device into no device at all. */ +static VkResult +wrapper_query_reset_reserve_queue(struct wrapper_device *device, + VkDeviceCreateInfo *info, + VkDeviceQueueCreateInfo **queues_out, + float **priorities_out) +{ + struct wrapper_physical_device *physical = device->physical; + uint32_t count = 0; + physical->dispatch_table.GetPhysicalDeviceQueueFamilyProperties( + physical->dispatch_handle, &count, NULL); + VkQueueFamilyProperties *families = calloc(count, sizeof(*families)); + if (!families) + return VK_ERROR_OUT_OF_HOST_MEMORY; + physical->dispatch_table.GetPhysicalDeviceQueueFamilyProperties( + physical->dispatch_handle, &count, families); + + /* vkCmdResetQueryPool needs graphics or compute. flags and pNext are + * excluded because wrapper_create_device_queue routes those through + * GetDeviceQueue2, and the shared-queue fallback compares raw handles. */ + int shareable = -1; + for (uint32_t i = 0; i < info->queueCreateInfoCount; i++) { + const VkDeviceQueueCreateInfo *q = &info->pQueueCreateInfos[i]; + uint32_t family = q->queueFamilyIndex; + if (q->flags || q->pNext || family >= count || !q->queueCount || + !(families[family].queueFlags & + (VK_QUEUE_GRAPHICS_BIT | VK_QUEUE_COMPUTE_BIT))) + continue; + if (shareable < 0) + shareable = i; + + uint32_t requested = 0; + for (uint32_t j = 0; j < info->queueCreateInfoCount; j++) + if (info->pQueueCreateInfos[j].queueFamilyIndex == family) + requested += info->pQueueCreateInfos[j].queueCount; + if (requested >= families[family].queueCount) + continue; + + VkDeviceQueueCreateInfo *queues = + malloc(info->queueCreateInfoCount * sizeof(*queues)); + float *priorities = malloc((q->queueCount + 1) * sizeof(*priorities)); + if (!queues || !priorities) { + free(queues); + free(priorities); + free(families); + return VK_ERROR_OUT_OF_HOST_MEMORY; + } + memcpy(queues, info->pQueueCreateInfos, + info->queueCreateInfoCount * sizeof(*queues)); + memcpy(priorities, q->pQueuePriorities, q->queueCount * sizeof(*priorities)); + priorities[q->queueCount] = 0.5f; + queues[i].queueCount++; + queues[i].pQueuePriorities = priorities; + device->query_reset_queue_family = family; + device->query_reset_queue_index = q->queueCount; + device->query_reset_shared_queue = false; + info->pQueueCreateInfos = queues; + *queues_out = queues; + *priorities_out = priorities; + free(families); + return VK_SUCCESS; + } + + if (shareable >= 0) { + device->query_reset_queue_family = + info->pQueueCreateInfos[shareable].queueFamilyIndex; + device->query_reset_queue_index = 0; + device->query_reset_shared_queue = true; + free(families); + WRAPPER_LOG(info, "No spare queue for vkResetQueryPool emulation, " + "sharing the application queue in family %u", + device->query_reset_queue_family); + return VK_SUCCESS; + } + + free(families); + WRAPPER_LOG(error, "vkResetQueryPool emulation requires a graphics or compute queue"); + return VK_ERROR_INITIALIZATION_FAILED; +} + +static VkResult +wrapper_query_reset_init(struct wrapper_device *device) +{ + const struct vk_device_dispatch_table *dt = &device->dispatch_table; + VkDevice dev = device->dispatch_handle; + dt->GetDeviceQueue(dev, device->query_reset_queue_family, + device->query_reset_queue_index, &device->query_reset_queue); + + VkResult result = dt->CreateCommandPool(dev, &(VkCommandPoolCreateInfo) { + .sType = VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO, + .flags = VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT | + VK_COMMAND_POOL_CREATE_TRANSIENT_BIT, + .queueFamilyIndex = device->query_reset_queue_family, + }, NULL, &device->query_reset_pool); + if (result != VK_SUCCESS) + return result; + + result = dt->AllocateCommandBuffers(dev, &(VkCommandBufferAllocateInfo) { + .sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO, + .commandPool = device->query_reset_pool, + .level = VK_COMMAND_BUFFER_LEVEL_PRIMARY, + .commandBufferCount = 1, + }, &device->query_reset_cmd); + if (result != VK_SUCCESS) + return result; + + result = dt->CreateFence(dev, &(VkFenceCreateInfo) { + .sType = VK_STRUCTURE_TYPE_FENCE_CREATE_INFO, + }, NULL, &device->query_reset_fence); + if (result == VK_SUCCESS) + WRAPPER_LOG(info, "Emulating vkResetQueryPool on %s queue (family=%u index=%u)", + device->query_reset_shared_queue ? "the application's" : "a private", + device->query_reset_queue_family, device->query_reset_queue_index); + return result; +} + +static void +wrapper_query_reset_finish(struct wrapper_device *device) +{ + if (device->query_reset_fence) + device->dispatch_table.DestroyFence(device->dispatch_handle, + device->query_reset_fence, NULL); + if (device->query_reset_pool) + device->dispatch_table.DestroyCommandPool(device->dispatch_handle, + device->query_reset_pool, NULL); +} + +VKAPI_ATTR void VKAPI_CALL +wrapper_ResetQueryPool(VkDevice _device, VkQueryPool queryPool, + uint32_t firstQuery, uint32_t queryCount) +{ + VK_FROM_HANDLE(wrapper_device, device, _device); + const struct vk_device_dispatch_table *dt = &device->dispatch_table; + VkResult result; + + if (device->physical->base_supported_features.hostQueryReset) { + dt->ResetQueryPool(device->dispatch_handle, queryPool, firstQuery, queryCount); + return; + } + /* Not emulating and the driver has no host reset: the application cannot have + * enabled the feature, because it was never advertised, so this is + * unreachable through valid usage. There is no entry point to forward to + * either -- dispatch_table.ResetQueryPool is NULL on such a driver. */ + if (!queryCount || !device->query_reset_queue) + return; + + simple_mtx_lock(&device->query_reset_mutex); + if (vk_device_is_lost(&device->vk)) { + simple_mtx_unlock(&device->query_reset_mutex); + return; + } + result = dt->ResetCommandBuffer(device->query_reset_cmd, 0); + if (result != VK_SUCCESS) + goto fail; + result = dt->BeginCommandBuffer(device->query_reset_cmd, + &(VkCommandBufferBeginInfo) { + .sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO, + .flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT, + }); + if (result != VK_SUCCESS) + goto fail; + dt->CmdResetQueryPool(device->query_reset_cmd, queryPool, firstQuery, queryCount); + result = dt->EndCommandBuffer(device->query_reset_cmd); + if (result != VK_SUCCESS) + goto fail; + result = dt->ResetFences(device->dispatch_handle, 1, &device->query_reset_fence); + if (result != VK_SUCCESS) + goto fail; + result = dt->QueueSubmit(device->query_reset_queue, 1, &(VkSubmitInfo) { + .sType = VK_STRUCTURE_TYPE_SUBMIT_INFO, + .commandBufferCount = 1, + .pCommandBuffers = &device->query_reset_cmd, + }, device->query_reset_fence); + if (result != VK_SUCCESS) + goto fail; + result = dt->WaitForFences(device->dispatch_handle, 1, + &device->query_reset_fence, VK_TRUE, UINT64_MAX); + if (result != VK_SUCCESS) + goto fail; + simple_mtx_unlock(&device->query_reset_mutex); + return; + +fail: + vk_device_set_lost(&device->vk, "Host query reset emulation failed: %d", result); + simple_mtx_unlock(&device->query_reset_mutex); +} + +VKAPI_ATTR VkResult VKAPI_CALL +wrapper_DeviceWaitIdle(VkDevice _device) +{ + VK_FROM_HANDLE(wrapper_device, device, _device); + if (!device->query_reset_queue) + return device->dispatch_table.DeviceWaitIdle(device->dispatch_handle); + + simple_mtx_lock(&device->query_reset_mutex); + VkResult result = vk_device_is_lost(&device->vk) ? VK_ERROR_DEVICE_LOST : + device->dispatch_table.DeviceWaitIdle(device->dispatch_handle); + simple_mtx_unlock(&device->query_reset_mutex); + return result; +} + static VkResult wrapper_create_device_queue(struct wrapper_device *device, const VkDeviceCreateInfo* pCreateInfo) @@ -632,6 +858,8 @@ wrapper_CreateDevice(VkPhysicalDevice physicalDevice, .sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FAULT_FEATURES_EXT, }; bool used_fallback_create = false; + VkDeviceQueueCreateInfo *query_reset_queues = NULL; + float *query_reset_priorities = NULL; device = vk_zalloc2(&physical_device->instance->vk.alloc, pAllocator, sizeof(*device), 8, VK_SYSTEM_ALLOCATION_SCOPE_DEVICE); @@ -649,6 +877,7 @@ wrapper_CreateDevice(VkPhysicalDevice physicalDevice, simple_mtx_init(&device->resource_mutex, mtx_plain); simple_mtx_init(&device->bcn_gpu_mutex, mtx_plain); + simple_mtx_init(&device->query_reset_mutex, mtx_plain); device->bcn_gpu_state = 0; device->physical = physical_device; @@ -742,6 +971,18 @@ if (pdf2 && pdf2->features.f) { \ wrapper_safe_create_device = getenv("WRAPPER_SAFE_CREATE_DEVICE") ? atoi(getenv("WRAPPER_SAFE_CREATE_DEVICE")) : 1; } + const bool emulate_query_reset = device->vk.enabled_features.hostQueryReset && + !physical_device->base_supported_features.hostQueryReset; + if (emulate_query_reset) { + result = wrapper_query_reset_reserve_queue(device, &wrapper_create_info, + &query_reset_queues, + &query_reset_priorities); + if (result != VK_SUCCESS) { + wrapper_DestroyDevice(wrapper_device_to_handle(device), &device->vk.alloc); + return vk_error(physical_device, result); + } + } + result = physical_device->dispatch_table.CreateDevice( physical_device->dispatch_handle, &wrapper_create_info, pAllocator, &device->dispatch_handle); @@ -757,6 +998,8 @@ if (pdf2 && pdf2->features.f) { \ } if (result != VK_SUCCESS) { + free(query_reset_queues); + free(query_reset_priorities); WRAPPER_LOG(error, "Failed driver createDevice, res %d", result); wrapper_emit_diag(physical_device, pCreateInfo, result); wrapper_DestroyDevice(wrapper_device_to_handle(device), @@ -765,11 +1008,22 @@ if (pdf2 && pdf2->features.f) { \ } } + free(query_reset_queues); + free(query_reset_priorities); + void *gdpa = physical_device->instance->dispatch_table.GetInstanceProcAddr( physical_device->instance->dispatch_handle, "vkGetDeviceProcAddr"); vk_device_dispatch_table_load(&device->dispatch_table, gdpa, device->dispatch_handle); + if (emulate_query_reset) { + result = wrapper_query_reset_init(device); + if (result != VK_SUCCESS) { + wrapper_DestroyDevice(wrapper_device_to_handle(device), &device->vk.alloc); + return vk_error(physical_device, result); + } + } + /* The fallback create with a NULL pNext drops the deviceFault feature, so * only treat fault reporting as usable when the primary create succeeded. */ device->device_fault_enabled = enable_device_fault && !used_fallback_create; @@ -1839,6 +2093,9 @@ wrapper_QueueSubmit(VkQueue _queue, uint32_t submitCount, const VkSubmitInfo* pSubmits, VkFence fence) { VK_FROM_HANDLE(wrapper_queue, queue, _queue); + if (queue->device->query_reset_queue && vk_device_is_lost(&queue->device->vk)) + return VK_ERROR_DEVICE_LOST; + const bool serialise = wrapper_query_reset_owns_queue(queue); VkSubmitInfo wrapper_submits[submitCount]; VkCommandBuffer *command_buffers; VkResult result; @@ -1860,8 +2117,12 @@ wrapper_QueueSubmit(VkQueue _queue, uint32_t submitCount, wrapper_submits[i].pCommandBuffers = command_buffers; } + if (serialise) + simple_mtx_lock(&queue->device->query_reset_mutex); result = queue->device->dispatch_table.QueueSubmit( queue->dispatch_handle, submitCount, wrapper_submits, fence); + if (serialise) + simple_mtx_unlock(&queue->device->query_reset_mutex); if (result == VK_ERROR_DEVICE_LOST) wrapper_log_device_fault(queue->device); @@ -1877,6 +2138,9 @@ wrapper_QueueSubmit2(VkQueue _queue, uint32_t submitCount, const VkSubmitInfo2* pSubmits, VkFence fence) { VK_FROM_HANDLE(wrapper_queue, queue, _queue); + if (queue->device->query_reset_queue && vk_device_is_lost(&queue->device->vk)) + return VK_ERROR_DEVICE_LOST; + const bool serialise = wrapper_query_reset_owns_queue(queue); VkSubmitInfo2 wrapper_submits[submitCount]; VkCommandBufferSubmitInfo *command_buffers; VkResult result; @@ -1898,8 +2162,12 @@ wrapper_QueueSubmit2(VkQueue _queue, uint32_t submitCount, wrapper_submits[i].pCommandBufferInfos = command_buffers; } + if (serialise) + simple_mtx_lock(&queue->device->query_reset_mutex); result = queue->device->dispatch_table.QueueSubmit2( queue->dispatch_handle, submitCount, wrapper_submits, fence); + if (serialise) + simple_mtx_unlock(&queue->device->query_reset_mutex); if (result == VK_ERROR_DEVICE_LOST) wrapper_log_device_fault(queue->device); @@ -2831,6 +3099,8 @@ wrapper_DestroyDevice(VkDevice _device, const VkAllocationCallbacks* pAllocator) { VK_FROM_HANDLE(wrapper_device, device, _device); + wrapper_query_reset_finish(device); + simple_mtx_lock(&device->resource_mutex); list_for_each_entry_safe(struct wrapper_command_buffer, wcb, @@ -2881,6 +3151,7 @@ wrapper_DestroyDevice(VkDevice _device, const VkAllocationCallbacks* pAllocator) simple_mtx_destroy(&device->push_mutex); } simple_mtx_destroy(&device->resource_mutex); + simple_mtx_destroy(&device->query_reset_mutex); vk_device_finish(&device->vk); vk_free2(&device->vk.alloc, pAllocator, device); } diff --git a/src/vulkan/wrapper/wrapper_physical_device.c b/src/vulkan/wrapper/wrapper_physical_device.c index efcd6d59..dfd93e7a 100644 --- a/src/vulkan/wrapper/wrapper_physical_device.c +++ b/src/vulkan/wrapper/wrapper_physical_device.c @@ -250,6 +250,25 @@ VkResult enumerate_physical_device(struct vk_instance *_instance) const uint32_t engine_version = instance->vk.app_info.engine_version; const uint32_t driver_version = pdevice->properties2.properties.driverVersion; + /* DXVK forces hostQueryReset on from D3D_FEATURE_LEVEL_9_1 up, so a driver + * without it gets no D3D11 device at any level. Gated on the driver + * rather than on the engine name: the base-lacks guard already limits + * this to drivers that need it, and the default is on only for the driver + * it was validated against. WRAPPER_EMULATE_HOST_QUERY_RESET is the + * tri-state override -- unset follows the driver default, true opts + * another driver in, false is the kill switch. */ + const bool nvidia_proprietary = + pdevice->driver_properties.driverID == VK_DRIVER_ID_NVIDIA_PROPRIETARY; + + if (!pdevice->base_supported_features.hostQueryReset && + debug_get_bool_option("WRAPPER_EMULATE_HOST_QUERY_RESET", + nvidia_proprietary)) { + WRAPPER_LOG(info, "Emulating VK_EXT_host_query_reset " + "(base driver lacks it; resets go through a command buffer)"); + pdevice->vk.supported_extensions.EXT_host_query_reset = true; + supported_features->hostQueryReset = true; + } + /* HACK: Specific prop drivers workarounds for Adreno and Mali GPUs */ if (pdevice->driver_properties.driverID == VK_DRIVER_ID_QUALCOMM_PROPRIETARY) { diff --git a/src/vulkan/wrapper/wrapper_private.h b/src/vulkan/wrapper/wrapper_private.h index 9ec85ec3..815c1cd1 100644 --- a/src/vulkan/wrapper/wrapper_private.h +++ b/src/vulkan/wrapper/wrapper_private.h @@ -108,6 +108,16 @@ struct wrapper_device { VkPipelineLayout bcn_pipe_layout; VkPipeline bcn_pipeline; VkDeviceSize bcn_gpu_inflight; /* transient GPU-transcode bytes not yet freed */ + + /* Private queue and synchronous command buffer for host query reset. */ + simple_mtx_t query_reset_mutex; + VkCommandPool query_reset_pool; + VkCommandBuffer query_reset_cmd; + VkFence query_reset_fence; + VkQueue query_reset_queue; + uint32_t query_reset_queue_family; + uint32_t query_reset_queue_index; + bool query_reset_shared_queue; /* no spare queue: shares the app's */ }; VK_DEFINE_HANDLE_CASTS(wrapper_device, vk.base, VkDevice,