Add TRT plugin custom ops support#13
Open
chilo-ms wants to merge 5 commits into
Open
Conversation
Register TensorRT plugins as custom ops under the 'trt.plugins' domain, enabling ORT to recognize and route nodes with TRT plugin op types to the TensorRT EP. Implementation: - Add TensorRTCustomOp/TensorRTCustomKernel structs using Ort::CustomOpBase with variadic inputs/outputs (TRT plugin API doesn't expose I/O counts) - Add CreateTensorRTCustomOpDomainList() that initializes default TRT plugins, loads extra plugin libraries, enumerates all registered plugin creators (V1 and V3One), and creates custom ops in the 'trt.plugins' domain - Wire GetNumCustomOpDomains/GetCustomOpDomains on OrtEpFactory - Add unit test adapted from upstream TRTPluginsCustomOpTest Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
|
Use DISCOVERY_MODE PRE_TEST so test discovery happens at ctest runtime instead of build time. This avoids exit code 0xc0000135 (DLL not found) when onnxruntime.dll and CUDA DLLs are not in PATH during the build step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Run 'Run unit tests' (trt_ep_tests) first - Rename 'Run tests' to 'Run onnxruntime_provider_test' for clarity Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The ORT build artifact may have onnxruntime.dll in a subdirectory. Use Get-ChildItem -Recurse to find it rather than assuming a fixed path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Register TensorRT plugins as custom ops under the
trt.pluginsdomain, enabling ORT to recognize and route nodes with TRT plugin op types to the TensorRT EP.Reference: upstream ORT implementation
Changes
New files
src/tensorrt_execution_provider_custom_ops.h- DefinesTensorRTCustomOp(usingOrt::CustomOpBase) andTensorRTCustomKernelstructs with variadic inputs/outputs, since TRT plugin API doesn't expose I/O counts.src/tensorrt_execution_provider_custom_ops.cc- ImplementsCreateTensorRTCustomOpDomainList():LoadLibraryA/dlopen)initLibNvInferPluginstrt.pluginsdomainModified files
src/tensorrt_provider_factory.h- AddedGetNumCustomOpDomainsImpl/GetCustomOpDomainsImplstatic methods andcustom_op_domain_list_member.src/tensorrt_provider_factory.cc- Wired the two function pointers in the constructor and implemented the static methods.Tests
tests/tensorrt_basic_test.cc- AddedTRTPluginsCustomOpTestadapted from the upstreamTensorrtExecutionProviderTest.TRTPluginsCustomOpTest. Validates that a model withtrt.pluginsdomain ops can be loaded and session creation succeeds.tests/testdata/trt_plugin_custom_op_test.py- Python script to generate the test model.tests/testdata/trt_plugin_custom_op_test.onnx- Generated test model withDisentangledAttention_TRTnode.