Multiple Image Pipeline nodes not compatible with multiple Argus Cameras

[Calvin-InDro]

Jetpack 6.0/6.1
x2 IMX219
x2 isaac_ros_argus_camera mono nodes
x2 isaac_ros_image_proc rectification nodes

While able to access any CSI connected IMX219s and stream multiple simultaneously via gst on Jetpack 6.0/6.1, opening multiple isaac_ros_argus_camera mono nodes simultaneously with different cameras results in strange behaviour from the isaac_ros_image_pipeline. When attempting to rectify the images for example, one stream (random if stream #1 or stream #2) will be rectified perfectly, while the other results in an empty image. Sometimes, however, the second stream appears to utilize nonsensical intrinsic camera parameters (while other times, rectifying completely normally). These are low-distortion lenses, and the camera calibrations are accurate, and DO result in perfectly rectified images when applied correctly (plum_bob models).

Pictured here are the rare times the second stream results in a non-empty image with nonsensical rectifications:

Any insight on this behavior would be greatly appreciate.

[rowanborder]

I am also experiencing this issue trying to use 4x IMX296 cameras with argus_camera and the rectify_node. 1 in 10 times all of the rectification nodes will start without issue, but the rest of the time at least one camera produces a black image or a corrupted image, as shown above.

[Infinite-Echo]

Also experiencing a similar issue, but with only one IMX219 camera. It is seemingly random whether it comes up correctly or not.

[sebastian-indro]

Same here, would love Jetpack 6.2 - isaac 3.2 support :)

[sgillen]

Hello, could one of you share the argus logs when this issue occurs?

sudo journalctl -u nvargus-daemon.service

[rowanborder]

I captured the argus logs from two attempts starting the camera driver node, nvidia::isaac_ros::argus::ArgusMonoNode, and the rectify node, nvidia::isaac_ros::image_proc::RectifyNode, - one where a black image was generated and the other where the image was correctly rectified. Unfortunately I can't notice a difference between the two.

I have also previously tried adding a delay between starting the camera driver node and the rectify node, in case there was some kind of race condition that started the rectification before the camera node was properly initialised, but this was unsuccessful.

I have also included the ISAAC ROS logs from the two attempts, in case those are useful, and a video illustrating the issue.

nvargus-daemon_rectify_correct_image.txt
nvargus-daemon_rectify_black_image.txt

isaac_ros_logs_rectify_correct_image.txt
isaac_ros_logs_rectify_black_image.txt

rectify_unreliability.mp4

[sebastian-indro]

@sgillen Hi again Sean, is this the info you needed? If something is missing we can provide more
Cheers thank you for helping on this issue!

[sgillen]

Hi, yes - @arai2601 could you please take a look at these argus logs?
Let's narrow down what layer of the stack the issue is occurring at.

[sgillen]

Arai got back to me offline, we don't see any argus level issues, and I don't see anything obvious from the ros level logs that were shared.

So lets try to narrow this down further, is it just the rectification that has issues or is it the output from the camera node? Are any of you able to replicate this issue without the rectification node? It may also help to get more verbose logs, which we can get with:

export GXF_LOG_LEVEL=info

[rowanborder]

I can confirm that the output from the camera node is correct in both scenarios. I'll try running the tests again with more verbose logging.

[rowanborder]

I ran the tests again with export GXF_LOG_LEVEL=info and have attached the logs. I don't notice any difference in the additional messages though.

isaac_ros_logs_rectify_correct_image_gxf_info.txt
isaac_ros_logs_rectify_black_image_gxf_info.txt