isaac_ros_image_proc fails to colcon build due to VPI/cupva error

[LR161204]

Hello,

I am having a problem with isaac_ros_image_proc using:

• A Jetson Orin Nano 8GB developer kit, which does not have PVA cores (cupva seems to be related to the cause of the error)
• L4T 35.4.1 and Jetpack 5.1.2 very recently flashed via Nvidia SDKmanager
• The Isaac ROS docker image, launched by run_dev.sh in isaac_ros_common
• VPI 2.3.9 on the host machine, I have confirmed that this is mounted into the docker container by run_dev.sh
• All Isaac ROS packages cloned since the most recent update, Isaac ROS 2.1 (build, install and log directories also deleted before colcon build)

Running colcon build in my isaac_ros-ws is able to build nearly every Isaac ROS package I have downloaded, including other packages that rely on VPI (all packages in isaac_ros_common, all packages in isaac_ros_nitros, isaac_ros_visual_slam, both packages in isaac_ros_object_detection and all the other packages in isaac_ros_image_pipeline). However it fails to build isaac_ros_image_proc with the following error:

Starting >>> isaac_ros_image_proc
[Processing: isaac_ros_image_proc]                            
[Processing: isaac_ros_image_proc]                                    
[Processing: isaac_ros_image_proc]                                       
[Processing: isaac_ros_image_proc]                                       
[Processing: isaac_ros_image_proc]                                       
[Processing: isaac_ros_image_proc]                                       
[Processing: isaac_ros_image_proc]                                       
[Processing: isaac_ros_image_proc]                                       
[Processing: isaac_ros_image_proc]                                       
--- stderr: isaac_ros_image_proc                                          
/usr/bin/ld: warning: libcupva_host.so.2.3, needed by /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libcupva_host_utils.so.2.3, needed by /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9, not found (try using -rpath or -rpath-link)
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::granularity(cupva::GranType)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::dstDim1(int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::ConfigDataFlow::id() const'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::RasterDataFlow::srcImpl(void const*, int, int, int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Fence::timestamp() const'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Fence::Fence(cupva::SyncObj&)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::RasterDataFlow::roi(int, int, int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdWaitOnFences::~CmdWaitOnFences()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Fence::wait(long) const'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::RasterDataFlow::dstImpl(void*, int, int, int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Executable::~Executable()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::handler(cupva::Parameter const&)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Context::~Context()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::SyncObj::Create(bool, cupva::SyncClientType, cupva::SyncWaitMode)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdProgram::CmdProgram()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Stream::~Stream()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::DynamicDataFlow::Node::bpp(int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::RasterDataFlow::link(cupva::RasterDataFlow&)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Exception::getErrorCode() const'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::DynamicDataFlow::~DynamicDataFlow()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Context::Context()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::tile(int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdWaitOnFences::CmdWaitOnFences(cupva::Fence const&)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::GetHardwareInfo()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::dstLinePitch(int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdProgram::Create(cupva::Executable const&)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::RasterDataFlow::handler(cupva::Parameter const&)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Parameter::setValuePointer(void*)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::ConfigDataFlow::~ConfigDataFlow()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdProgram::operator=(cupva::CmdProgram&&) &'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::padVal(cupva::PadModeType, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::RasterDataFlow::halo(int, int, cupva::PadModeType, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Stream::submit(std::initializer_list<cupva::BaseCmd const*> const&, cupva::impl::CmdStatus**, cupva::OrderType, int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdProgram::compileDataFlows()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::DynamicDataFlow::Create()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Parameter::setValueArray(void const*, long)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::mem::Alloc(long, cupva::mem::AccessType, cupva::mem::AllocType)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Executable::Executable(cupva::Executable&&)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Context::Create(unsigned int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdProgram::~CmdProgram()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::src(void const*)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::RasterDataFlow::tile(int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdProgram::Create()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::DynamicDataFlow::at(int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::mem::GetSurfaceAttributes(void const*, cupva::mem::SurfaceAttributes&)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::srcCircularBuffer(int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdRequestFences::~CmdRequestFences()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::srcLinePitch(int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::ConfigDataFlow::Create()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Stream::submit(cupva::BaseCmd const* const*, cupva::impl::CmdStatus**, int, cupva::OrderType, int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Context::GetCurrent()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::RasterDataFlow::tileBufferImpl(void*, void*)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Parameter::Parameter()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdProgram::registerDataFlowHead(cupva::BaseDataFlow&&, int, float)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Parameter::getDevicePointer() const'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::ConfigDataFlow::linkInternal(cupva::BaseDataFlow&, bool)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Fence::~Fence()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::DynamicDataFlow::Node::src(void const*, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdProgram::registerDataFlow(cupva::BaseDataFlow&&)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::dstCircularBuffer(int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Parameter::~Parameter()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva_utils::AllocSurface(cupva_utils::PlaneSize const*, int, cupva::SurfaceFormatType)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::DynamicDataFlow::Node::dst(void*, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::RasterDataFlow::~RasterDataFlow()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::dstDim2(int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdRequestFences::CmdRequestFences(cupva::Fence&)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Context::operator=(cupva::Context&&) &'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::ConfigDataFlow::handler(cupva::Parameter const&)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::SyncObj::~SyncObj()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Context::SetCurrent(cupva::impl::Context*)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::dst(void*, void*)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::mem::Free(void*)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::RasterDataFlow::Create()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Executable::Create(void const*, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::mem::GetHostPointer(void*)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::DynamicDataFlow::Node::tile(int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::DynamicDataFlow::init(cupva::Parameter const&, int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::CmdProgram::operator[](char const*)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::linkInternal(cupva::BaseDataFlow&, bool)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::RasterDataFlow::halo(int, cupva::PadModeType, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::padDim(cupva::PadDirType, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::~StaticDataFlow()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::bpp(int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::ConfigDataFlow::src(void const*)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::srcDim1(int, int)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::id() const'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Parameter::setValueScalar(void const*, long)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::Create()'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::Stream::Create(cupva::EngineType, cupva::AffinityType)'
/usr/bin/ld: /opt/nvidia/vpi2/lib/aarch64-linux-gnu/libnvvpi.so.2.3.9: undefined reference to `cupva::StaticDataFlow::srcDim2(int, int)'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/isaac_ros_image_proc.dir/build.make:367: isaac_ros_image_proc] Error 1
make[1]: *** [CMakeFiles/Makefile2:310: CMakeFiles/isaac_ros_image_proc.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
---
Failed   <<< isaac_ros_image_proc [4min 54s, exited with code 2]

Summary: 0 packages finished [4min 55s]
  1 package failed: isaac_ros_image_proc
  1 package had stderr output: isaac_ros_image_proc

I would appreciate any advice on how to stop VPI from trying to use cupva, or any indication as to what is causing the problem with the most recent version of isaac_ros_image_pipeline for Isaac ROS 2.1 on a Jetson Orin Nano.