docs: forward RGB camera over RTSP — the grey frame is not a covered lens, and how to view it without costing a CPU core

[armwaheed]

@CcShuo — flagging you directly. This comes out of the DR02 Pro session you were present for on 3 September. I tried to assign the PR to you but could not: as an outside contributor I do not have permission to set assignees on this repository, so this mention is the substitute. Please reassign or redirect it if someone else is the right reviewer.

@CcShuo —— 直接 @ 你。本 PR 来自 9 月 3 日你在场的那次 DR02 Pro 实机调试。我尝试把 PR 指派给你但没有成功：作为外部贡献者，我没有在本仓库设置 assignee 的权限，因此以 @ 提及代替。如果应由其他同事评审，请随时转派。

---

English | 简体中文

Why this matters

An operator standing on a factory floor next to a DR02 Pro needs to see what the robot sees. It turns out that both obvious ways of doing that have a trap in them: the first one makes a working camera look broken, and the second one quietly costs the robot most of a CPU core.

We hit both during a bring-up session on a DR02 Pro on 3 September 2026, and the first one cost us several hours and a wrong conclusion. This PR documents what we learned so the next person does not spend that time. None of it is a defect report — the behaviour is reasonable in every case, it is just invisible from the outside.

What we found

1. The forward camera is not a RealSense, and publishes no ROS 2 Topic.

docs/CAMERA.md documents three RealSense D435 units on the NOS host. On the robot we measured, all three were mounted at a downward angle, and none of them gave a view along the direction of travel. The forward view came from a separate RGB camera on the AOS host, served as an RTSP stream at rtsp://10.21.33.103:8554/video1. It is not opened by realsense2_camera and it publishes nothing on DDS. video0 does not open.

We could not find this written down anywhere, so we spent a while looking for a Topic that does not exist.

2. Opened naively, it returns a uniform grey frame that looks exactly like a covered lens.

This is the part worth reading. Opened with OpenCV's defaults the stream connects, correctly reports 1280×720, and hands back a flat grey image — and cap.read() returns True for every one of those frames.

We reported this camera as having a covered lens. It did not. The camera was working perfectly the whole time. That correction is the most useful thing in this PR.

The cause is visible in VLC's own log on the same stream, which repeats hevc packetizer: Waiting for VPS/SPS/PPS. The stream is H.265, and its parameter sets are sent infrequently rather than ahead of every keyframe. Until one lands, no decoder anywhere can draw a picture — VLC included, which just hides the gap behind a black window. Over UDP, lost fragments make the gap longer.

Two things fix it. Force TCP before the capture is opened:

os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "rtsp_transport;tcp|max_delay;5000000"

And judge a frame by its variance, not by whether the read succeeded. Measured on one capture:

Frame	Standard deviation	Laplacian variance	Reading
0	1.06	6.2	Uniform grey; cap.read() returned True
20 onward	≈68	≈330	A picture

Those two populations are an order of magnitude apart, so the threshold does not need tuning. A program that treats a successful read as a valid image will report a working camera as a broken one, which is exactly what we did.

3. The RTSP server appears to serve one client at a time. A second consumer evicted the first with 454 Session Not Found. A forgotten ffplay or VLC window on a development host is enough to kill an application's stream for no visible reason.

4. Doing it responsibly is the other half of the problem.

A monitoring view that displayed all four cameras made itself the largest single CPU consumer on the robot — larger than the LiDAR driver — at 91% of one core. A view an operator uses to watch the robot should not be competing with the robot's own perception stack for the machine it is standing on.

Two changes fixed it, both the same principle: do not pay for pixels nobody is looking at.

• Release the capture entirely when nobody is watching. Reading and discarding still decodes H.265 at 720p — the decode is the cost, not the display. Skipping only the encode took the process from 121% to 91% of one core, which was not enough. Only closing the stream stops it.
• For the ROS 2 cameras, destroy the subscription rather than filtering in the callback. rclpy deserialises a message before the callback runs, so an early return has already paid in full. Three 640×480 streams at 30 Hz is roughly 83 MB/s of pixels copied into Python for images nobody is looking at, and an early return saves none of it. The same measurement on /LIDAR/POINTS makes the size of the effect clear: a reader held open cost 4.0% of one core doing nothing but deserialising and discarding sweeps; created, used for one sweep, and destroyed again, it was 0.1%.

Measured result:

Configuration	CPU
All four views open; only the encode skipped when unwatched	121% of one core
Encode skipped, captures and subscriptions still held open	91% of one core
Captures released and subscriptions destroyed; no view open	3.26% of one core
Captures released and subscriptions destroyed; one view open	6.72% of one core

What is in this PR

File	Change
src/dr02_pro/docs/FORWARD_CAMERA.md	New. The forward RGB camera: stream address, TCP transport, the grey-frame check, one-client-at-a-time, the CPU discipline, a minimal reader, lens characteristics, troubleshooting.
src/dr02_pro/docs/FORWARD_CAMERA_CN.md	New. Full Chinese translation of the above.
src/dr02_pro/docs/CAMERA.md / CAMERA_CN.md	One scope paragraph: this document covers the three RealSense units; the forward camera is elsewhere.
src/dr02_pro/README.md / README_CN.md	One row in the documentation table, one bullet in the feature list.

Documentation only — no code, no build changes, and dr02_std is untouched. The new document follows the shape of your existing CAMERA.md: the same back-link, the same host names and addresses (10.21.33.103 / 10.21.33.106), the same table and callout conventions, and an English/_CN pair.

On an example. The working reader is included inline in the document rather than as a dr02_pro example, because building it would add an OpenCV dependency to a package whose examples currently need only rclcpp and drdds. That felt like the wrong trade to make on your behalf. If you would rather have it as a built example — or as a small node that republishes the RTSP stream as a ROS 2 Topic, which would also solve the one-client problem — we are happy to add it.

Scope, and what we could not verify

Everything above was measured on one DR02 Pro on 3 September 2026. One robot, one day. Host addresses, the stream path, and the camera mounting angles are that robot's. The shape of each behaviour should generalise; the specific numbers are not promised to, and the document says so at the top.

Three things we want to flag explicitly rather than let stand as if they were solid:

• The CPU percentages were read from top during the session and not recorded to a file. They are the size of an effect, not precise figures. The document says this.
• The one-client-at-a-time behaviour was observed, not characterised. We saw 454 Session Not Found when a second consumer connected; we did not test whether it is a hard limit, a configuration, or a timeout.
• We did not establish the relationship between Developer Mode and the RTSP service. We read the stream without changing modes, but did not test this systematically. If Developer Mode does affect it, that is worth a line in the document and we will add it.

Corrections on any of these are very welcome — particularly if the mounting angles differ across units, since that would change how the first paragraph should be worded.

Thank you for the session, and for the time on the floor. The grey-frame trap in particular is genuinely subtle — it is the decoder behaving correctly, and there is no reasonable way the documentation could have anticipated that a successful read() would be the misleading part. We would rather it be written down than not.

---

简体中文

English | 简体中文

为什么这件事重要

站在产线现场、就在 DR02 Pro 旁边的操作员，需要能看到机器人看到的画面。而做这件事的两条最直观的路径各自都有一个坑：第一个坑会让一台工作正常的相机看起来像坏了，第二个坑会悄悄吃掉机器人将近一个 CPU 核。

我们在 2026 年 9 月 3 日的一次 DR02 Pro 实机调试中把这两个坑都踩了一遍，其中第一个花掉了我们几个小时，并且得出了一个错误结论。本 PR 把我们学到的东西写下来，让下一个人不必再花这些时间。这些内容都不是缺陷报告——每一处行为本身都是合理的，只是从外部完全看不出来。

我们发现了什么

1. 前向相机不是 RealSense，也不发布任何 ROS 2 Topic。

docs/CAMERA_CN.md 记录了 NOS 主机上的三台 RealSense D435。在我们实测的那台机器人上，这三台相机都是向下倾斜安装的，没有任何一台提供沿行进方向的视野。前向视野来自 AOS 主机上的另一台独立 RGB 相机，以 RTSP 视频流的形式提供，地址为 rtsp://10.21.33.103:8554/video1。它不由 realsense2_camera 打开，在 DDS 上也不发布任何内容。video0 无法打开。

我们没有在任何地方找到这一说明，因此花了不少时间去寻找一个并不存在的 Topic。

2. 直接打开时返回的是一整块灰色图像，看起来与镜头被遮挡完全一样。

这一部分最值得一读。使用 OpenCV 的默认参数打开时，视频流能够正常连接，也能正确报告 1280×720 的分辨率，但返回的是一片平坦的灰色图像——而且 cap.read() 对其中每一帧都返回 True。

我们当时把这台相机报告为镜头被遮挡。事实并非如此，它自始至终工作正常。 这个更正是本 PR 中最有价值的内容。

原因可以从 VLC 自身在同一视频流上的日志中看到，它反复输出 hevc packetizer: Waiting for VPS/SPS/PPS。该视频流为 H.265 编码，其参数集的发送间隔较长，并不会在每个关键帧之前发送。在参数集到达之前，任何解码器都无法输出画面——VLC 也不例外，它只是用黑色窗口把这段时间遮住了。在 UDP 传输下，分片丢失会让这段时间更长。

有两点可以解决问题。第一，在打开采集之前强制使用 TCP：

os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "rtsp_transport;tcp|max_delay;5000000"

第二，应根据图像方差判断，而不是根据读取是否成功判断。 一次采集中的实测数据：

帧号	标准差	拉普拉斯方差	判读
0	1.06	6.2	一致灰色；cap.read() 返回 True
20 及之后	≈68	≈330	正常图像

两组数值相差一个数量级，因此阈值无需精细调整。把读取成功当作图像有效的程序，会把一台工作正常的相机报告为故障相机——我们当时正是这样做的。

3. RTSP 服务表现为同一时间只服务一个客户端。 第二个客户端接入会把第一个踢出，并报 454 Session Not Found。开发主机上一个被遗忘的 ffplay 或 VLC 窗口，就足以让应用程序的视频流毫无征兆地中断。

4. 问题的另一半，是如何不给机器人添负担。

一个同时显示四路相机的监控页面，成为了机器人上占用 CPU 最多的单个负载——超过了激光雷达驱动——达到一个核的 91%。操作员用来观察机器人的页面，不应该与机器人自身的感知程序争抢它正站立其上的这台机器的算力。

有两项改动解决了这个问题，二者遵循同一个原则：不要为没有人在看的图像付出代价。

• 无人查看时应彻底释放采集句柄。 读取后丢弃仍然会在 720p 下完整解码 H.265——解码才是开销所在，显示不是。仅跳过编码只把进程从一个核的 121% 降到 91%，并不够，只有关闭视频流才能真正停止开销。
• 对于 ROS 2 相机，应销毁订阅，而不是在回调中过滤。 rclpy 会在回调执行之前完成反序列化，因此提前 return 时代价已经完整付出。三路 640×480 图像以 30 Hz 发布，约相当于每秒 83 MB 的像素数据被复制进 Python，而这些图像根本没有人在看，提前 return 一点也省不下来。在 /LIDAR/POINTS 上的同一测量可以更清楚地看出量级：一直保持打开的订阅，仅仅是反序列化并丢弃点云就占用了一个核的 4.0%；改为每次只为一帧点云创建订阅、用完即销毁后，结果是 0.1%。

实测结果：

配置	CPU
四路视图全开；无人查看时仅跳过编码	一个核的 121%
跳过编码，但采集句柄与订阅仍保持打开	一个核的 91%
释放采集句柄并销毁订阅；未打开任何视图	一个核的 3.26%
释放采集句柄并销毁订阅；打开一路视图	一个核的 6.72%

本 PR 的内容

文件	改动
src/dr02_pro/docs/FORWARD_CAMERA.md	新增。前向 RGB 相机：视频流地址、TCP 传输、灰色图像的判断方法、单客户端限制、CPU 开销控制、最小读取程序、镜头特性和常见问题。
src/dr02_pro/docs/FORWARD_CAMERA_CN.md	新增。上述文档的完整中文版。
src/dr02_pro/docs/CAMERA.md / CAMERA_CN.md	增加一段范围说明：本文只涉及三台 RealSense 相机，前向相机另见新文档。
src/dr02_pro/README.md / README_CN.md	文档导航表中增加一行，功能概览中增加一条。

仅为文档改动——不含代码，不改动编译配置，dr02_std 未作任何改动。新文档沿用了现有 CAMERA.md 的形式：相同的返回链接、相同的主机名称与地址（10.21.33.103 / 10.21.33.106）、相同的表格与提示块约定，以及英文 / _CN 成对的组织方式。

关于示例程序。 可运行的读取程序是以代码块的形式写在文档里的，而没有做成 dr02_pro 的编译示例，因为那样会给一个当前仅依赖 rclcpp 和 drdds 的软件包引入 OpenCV 依赖。我们认为这个取舍不应由我们替你们决定。如果你们更希望把它做成编译示例——或者做成一个把 RTSP 流转发为 ROS 2 Topic 的小节点，那样也能顺带解决单客户端的问题——我们很乐意补充。

适用范围，以及我们无法确认的部分

以上全部内容均在 2026 年 9 月 3 日于一台 DR02 Pro 上实测得到：一台机器人，一天时间。 主机地址、视频流路径和相机安装角度均为该机器人的实际情况。各类现象的形态应当具有普遍性，但具体数值不作保证，文档开头也已如此说明。

有三点我们希望明确标注出来，而不是让它们看起来像是已经确凿的结论：

• CPU 百分比是实测过程中从 top 读取的，没有记录到文件。 它们表示的是效应的量级，不是精确数值。文档中已说明这一点。
• 单客户端行为只是观察到的，并未系统表征。 我们看到第二个客户端接入时出现 454 Session Not Found，但没有验证这究竟是硬性限制、某项配置，还是超时导致的。
• 我们没有确定开发者模式与 RTSP 服务之间的关系。 我们当时未切换模式即可读取视频流，但这一点未经系统性验证。如果开发者模式确实会产生影响，值得在文档中加上一句，我们会补上。

以上任何一点如有指正，我们都非常欢迎——尤其是如果不同整机之间的相机安装角度存在差异，那么第一段的措辞就需要相应调整。

感谢这次实机调试，也感谢现场的时间。灰色图像这个坑确实很隐蔽——那是解码器在正确工作，而且没有任何合理的方式能让文档预先想到「read() 成功」才是真正误导人的地方。我们认为把它写下来，比不写下来要好。