From 301ec40d3f3098679d89a35f20a1766d09369812 Mon Sep 17 00:00:00 2001 From: armwaheed Date: Thu, 3 Sep 2026 20:35:58 +0800 Subject: [PATCH] docs: document the forward RGB camera (RTSP) and its CPU cost The three RealSense D435 units documented in CAMERA.md were, on the DR02 Pro measured, all angled downward. The only forward-facing camera was a separate RGB unit on the AOS host, served as an RTSP stream and publishing no ROS 2 Topic, which is not covered by any existing document. Adds docs/FORWARD_CAMERA.md and docs/FORWARD_CAMERA_CN.md covering: - the stream address, and that video0 does not open - forcing TCP transport before the capture is opened - why the first frames decode as uniform grey (H.265 parameter sets arrive infrequently) and why cap.read() returns True for them, so a frame must be judged by its variance rather than by whether the read succeeded - the server serving one client at a time - releasing the capture rather than reading and discarding, and destroying ROS 2 subscriptions rather than filtering in the callback, with the CPU measurements that motivated both Also adds a scope pointer from CAMERA.md/CAMERA_CN.md and index entries in the DR02 Pro README pair. Measured on one DR02 Pro on 3 September 2026. Co-Authored-By: Claude Opus 5 (1M context) --- src/dr02_pro/README.md | 2 + src/dr02_pro/README_CN.md | 2 + src/dr02_pro/docs/CAMERA.md | 2 + src/dr02_pro/docs/CAMERA_CN.md | 2 + src/dr02_pro/docs/FORWARD_CAMERA.md | 287 +++++++++++++++++++++++++ src/dr02_pro/docs/FORWARD_CAMERA_CN.md | 285 ++++++++++++++++++++++++ 6 files changed, 580 insertions(+) create mode 100644 src/dr02_pro/docs/FORWARD_CAMERA.md create mode 100644 src/dr02_pro/docs/FORWARD_CAMERA_CN.md diff --git a/src/dr02_pro/README.md b/src/dr02_pro/README.md index e1f64fc..c8d7c5e 100644 --- a/src/dr02_pro/README.md +++ b/src/dr02_pro/README.md @@ -12,6 +12,7 @@ This ROS 2 package provides the DR02 Pro motion-control state machine and standa - Monitoring examples for battery state and current-fault snapshots - Peripheral Topic examples for IMU and gamepad keys - RealSense depth-camera deployment and usage documentation +- Forward-facing RGB camera access over RTSP - Audio examples for WAV playback, volume control, and recording ## Directory Layout @@ -34,6 +35,7 @@ This ROS 2 package provides the DR02 Pro motion-control state machine and standa | [State Machine](docs/STATE_MACHINE.md) | State transitions, runtime commands, keyboard control, gamepad control, and safety requirements | | [Topic Examples](docs/EXAMPLES.md) | Topics, real-robot Developer Modes, simulation support, runtime commands, and notes for all examples | | [RealSense Cameras](docs/CAMERA.md) | Camera driver installation on the NOS host, ROS 2 driver usage, the librealsense C/C++ interface, the pyrealsense2 Python interface, three-camera startup, and Topic verification | +| [Forward RGB Camera](docs/FORWARD_CAMERA.md) | The forward-facing RGB camera on the AOS host: RTSP stream address, required transport settings, how to tell a decoded frame from an undecoded one, and how to keep camera viewing off the robot's CPU | | [Joint Control Interface](docs/JOINT_CONTROL.md) | Joint message fields, array order, controllable ranges, units, and zero-position definition | ## Quick Start diff --git a/src/dr02_pro/README_CN.md b/src/dr02_pro/README_CN.md index c27a025..7173b6f 100644 --- a/src/dr02_pro/README_CN.md +++ b/src/dr02_pro/README_CN.md @@ -12,6 +12,7 @@ - 电池状态和当前故障快照监测示例 - IMU 和手柄按键外设 Topic 示例 - RealSense 深度相机部署和使用说明 +- 前向 RGB 相机的 RTSP 访问方式 - WAV 文件播放、音量控制和录音示例 ## 目录结构 @@ -34,6 +35,7 @@ | [状态机](docs/STATE_MACHINE_CN.md) | 状态流转、运行命令、键盘控制、手柄控制和安全要求 | | [Topic 示例](docs/EXAMPLES_CN.md) | 全部示例的 Topic、实机开发者模式、仿真支持、运行命令和注意事项 | | [RealSense 相机](docs/CAMERA_CN.md) | NOS 主机上的相机驱动安装、ROS 2 驱动、librealsense C/C++ 接口、pyrealsense2 Python 接口、三相机启动和 Topic 验证 | +| [前向 RGB 相机](docs/FORWARD_CAMERA_CN.md) | AOS 主机上的前向 RGB 相机：RTSP 地址、必需的传输设置、如何区分已解码与未解码图像，以及如何避免查看相机占用机器人 CPU | | [关节控制接口](docs/JOINT_CONTROL_CN.md) | 关节消息字段、数组顺序、可控范围、单位和零位定义 | ## 快速开始 diff --git a/src/dr02_pro/docs/CAMERA.md b/src/dr02_pro/docs/CAMERA.md index 3512b51..878d31e 100644 --- a/src/dr02_pro/docs/CAMERA.md +++ b/src/dr02_pro/docs/CAMERA.md @@ -4,6 +4,8 @@ The DR02 Pro is equipped with three Intel RealSense D435 depth cameras that provide color and depth images for applications such as environmental perception, object recognition, and 3D vision. All three cameras connect through USB to the NOS host (`10.21.33.106`). Camera drivers and related programs should be installed and run on this device. +This document covers the three RealSense depth cameras only. The DR02 Pro also carries a forward-facing RGB camera attached to the AOS host, which is not a RealSense, is not opened by `realsense2_camera`, and publishes no ROS 2 Topic. The three RealSense units are mounted at fixed angles that may not include a view along the direction of travel; on at least one DR02 Pro all three were angled downward. If a forward view is required, see [Forward RGB Camera](FORWARD_CAMERA.md). + > [!NOTE] > > The RealSense driver deployment, ROS 2 driver usage, and librealsense SDK calls in this document follow the official methods. Packages, dependencies, launch parameters, and APIs may change between versions. Refer to the official [RealSense ROS 2 Wrapper](https://github.com/realsenseai/realsense-ros) and [librealsense SDK](https://github.com/realsenseai/librealsense) documentation for the latest requirements. DR02 Pro-specific operations are identified separately in the relevant steps. diff --git a/src/dr02_pro/docs/CAMERA_CN.md b/src/dr02_pro/docs/CAMERA_CN.md index 81a1662..ecc852c 100644 --- a/src/dr02_pro/docs/CAMERA_CN.md +++ b/src/dr02_pro/docs/CAMERA_CN.md @@ -4,6 +4,8 @@ DR02 Pro 配置三台 Intel RealSense D435 深度相机，可提供彩色图像和深度图像，为环境感知、目标识别和三维视觉等应用提供数据。三台相机均通过 USB 连接至 NOS 主机（`10.21.33.106`），相机驱动及相关程序应在该设备上安装和运行。 +本文只涉及三台 RealSense 深度相机。DR02 Pro 上还有一台挂在 AOS 主机上的前向 RGB 相机，它不是 RealSense 相机，不由 `realsense2_camera` 打开，也不发布任何 ROS 2 Topic。三台 RealSense 相机的安装角度固定，可能并不包含沿行进方向的视野；在至少一台 DR02 Pro 上，三台相机均向下倾斜安装。如果需要前向视野，请参阅[前向 RGB 相机](FORWARD_CAMERA_CN.md)。 + > [!NOTE] > > 本文中的 RealSense 驱动部署、ROS 2 驱动使用和 librealsense SDK 调用均采用官方提供的方式。软件包、依赖关系、启动参数和 API 可能随版本更新，最新要求以 [RealSense ROS 2 Wrapper](https://github.com/realsenseai/realsense-ros) 和 [librealsense SDK](https://github.com/realsenseai/librealsense) 官方文档为准。DR02 Pro 特有的产品操作将在对应步骤中单独说明。 diff --git a/src/dr02_pro/docs/FORWARD_CAMERA.md b/src/dr02_pro/docs/FORWARD_CAMERA.md new file mode 100644 index 0000000..76415be --- /dev/null +++ b/src/dr02_pro/docs/FORWARD_CAMERA.md @@ -0,0 +1,287 @@ +# DR02 Pro Forward RGB Camera Guide + +[Back to the DR02 Pro SDK Guide](../README.md) + +The three Intel RealSense D435 cameras described in [RealSense Cameras](CAMERA.md) connect to the NOS host (`10.21.33.106`) and publish ROS 2 Topics. On the robot used for the measurements below, all three were mounted at a downward angle, and none of them provided a forward view along the direction of travel. + +The forward view on that robot came from a separate RGB camera attached to the AOS host (`10.21.33.103`). That camera is not a RealSense, is not opened by `realsense2_camera`, and **publishes no ROS 2 Topic**. It is reached as an RTSP video stream. This document describes how to read that stream, how to tell a decoded picture from an undecoded one, and how to keep the stream from becoming the largest CPU consumer on the robot. + +> [!NOTE] +> +> Everything in this document was measured on a single DR02 Pro on 3 September 2026 — one robot, one day. Host addresses, stream paths, and camera mounting angles are that robot's. The *shape* of each behavior described here is expected to generalize; the specific values are not promised to. Confirm the mounting angles and the stream address on your own unit before relying on them. + +## Camera Access Methods + +| Camera | Host | Transport | Access method | +| --- | --- | --- | --- | +| 3 × RealSense D435 | NOS host (`10.21.33.106`) | USB | `realsense2_camera` ROS 2 driver, or the librealsense SDK. See [RealSense Cameras](CAMERA.md). | +| Forward RGB | AOS host (`10.21.33.103`) | RTSP over the robot network | Any RTSP client. No ROS 2 Topic is published. | + +The data path for the forward camera is: + +```text +camera -> RTSP server on the AOS host -> RTSP client (OpenCV, FFmpeg, VLC, GStreamer) +``` + +If the forward image is required by other ROS 2 nodes, the user program must read the RTSP stream and publish the messages itself. Nothing on the robot does this automatically. + +## Stream Address + +| Path | Result | +| --- | --- | +| `rtsp://10.21.33.103:8554/video1` | Serves the forward RGB stream at 1280×720. | +| `rtsp://10.21.33.103:8554/video0` | Does not open. | + +Confirm the stream from the command line before writing any code against it: + +```bash +# From a development host on the robot network, or on the AOS host itself. +ffprobe -rtsp_transport tcp -v error -show_streams \ + rtsp://10.21.33.103:8554/video1 +``` + +The codec reported is `hevc` — H.265. That single fact explains most of the behavior in the next section. + +## Reading the Stream + +### 1. Force TCP Before the Capture Is Opened + +OpenCV's FFmpeg backend defaults to UDP transport for RTSP. On this stream, UDP loses fragments during the interval before the decoder can produce a picture and extends that interval considerably. Set the transport through the environment variable that the FFmpeg backend reads. The variable must be set **before** the capture is opened; setting it before `import cv2` is the reliable order: + +```python +import os + +os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "rtsp_transport;tcp|max_delay;5000000" + +import cv2 + +cap = cv2.VideoCapture("rtsp://10.21.33.103:8554/video1", cv2.CAP_FFMPEG) +``` + +With `ffmpeg` or `ffprobe` directly, the equivalent is `-rtsp_transport tcp`. With VLC, it is `--rtsp-tcp`. + +### 2. Judge a Frame by Its Variance, Not by Whether the Read Succeeded + +> [!IMPORTANT] +> +> Opened with default options, this stream connects, correctly reports 1280×720, and hands back a **uniform grey frame**. `cap.read()` returns `True` for every one of those frames. A uniform grey frame from a camera looks exactly like a covered lens, and it was initially reported as one during the session that produced this document. The camera was working correctly the whole time. + +The cause is visible in VLC's own log on the same stream, which repeats: + +```text +hevc packetizer: Waiting for VPS/SPS/PPS +``` + +The stream is H.265/HEVC, and its parameter sets — VPS, SPS, and PPS — are transmitted infrequently rather than ahead of every keyframe. No decoder can produce a picture until one arrives. Every client shows grey during that window, including VLC, which hides it behind a black window until it has something to draw. A capture loop that takes the first frame `read()` returns to it therefore samples that window, and a program that treats a successful read as a valid image will report a working camera as a broken one. + +A frame's own statistics separate the two cases cleanly. Measured across one capture on this stream: + +| Frame | Standard deviation | Laplacian variance | Reading | +| --- | --- | --- | --- | +| 0 | 1.06 | 6.2 | Uniform grey. `cap.read()` returned `True`. | +| 20 onward | ≈68 | ≈330 | A picture. | + +The two populations are separated by more than an order of magnitude, so the threshold does not need to be tuned carefully: + +```python +import cv2 + + +def frame_is_decoded(frame, min_std: float = 10.0) -> bool: + """True once the decoder has received its parameter sets. + + A frame produced before VPS/SPS/PPS arrive is uniform grey, and read() + reports success for it. Measured on this stream: std 1.06 before, + approximately 68 after. Any threshold between the two works. + """ + gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) + return float(gray.std()) >= min_std +``` + +Laplacian variance (`cv2.Laplacian(gray, cv2.CV_64F).var()`) separates the same two populations and additionally reports focus, but the standard deviation is cheaper and is sufficient on its own to decide whether a frame is decoded. + +Discard frames until the check passes. On this robot the picture appeared at approximately frame 20; allow a generous bound rather than a fixed count, and report a timeout rather than publishing grey. + +### 3. One Client at a Time + +The RTSP server on this robot appeared to serve **one client at a time**. Opening a second consumer while the first was connected evicted the first, which then failed with: + +```text +454 Session Not Found +``` + +Plan for a single reader. If several consumers need the forward image, read the stream once and republish it — as a ROS 2 Topic, or over whatever transport the application already uses — rather than pointing every consumer at the RTSP URL. Note that a stray `ffplay`, `ffprobe`, or VLC window left open on a development host is a consumer, and is a common reason for an application's stream to die without an obvious cause. + +## Keeping the Cost Off the Robot + +Decoding H.265 at 1280×720 is not free, and the CPU it uses belongs to the robot. During the session that produced this document, a monitoring view that displayed all four cameras became the largest single CPU consumer on the robot — larger than the LiDAR driver — at **91% of one core**. An operator looking at a camera should not be competing with the robot's own perception stack for the machine it is standing on. + +Two changes fixed it. Both are the same principle: **do not pay for pixels nobody is looking at.** + +### Release the Capture, Do Not Read and Discard + +Reading frames and throwing them away still decodes every one of them. The decode is the cost; the encode and display that follow it are comparatively minor. Skipping only the encode while continuing to read took the process from 121% to 91% of one core, which was not enough. + +Closing the capture is what stops the cost: + +```python +class ForwardCamera: + """Holds the RTSP capture open only while someone is watching.""" + + URL = "rtsp://10.21.33.103:8554/video1" + + def __init__(self): + self.cap = None + + def open(self): + if self.cap is None: + self.cap = cv2.VideoCapture(self.URL, cv2.CAP_FFMPEG) + return self.cap.isOpened() + + def close(self): + # Not "stop encoding" and not "read and discard". The H.265 decode is + # the cost, and only releasing the capture stops it. + if self.cap is not None: + self.cap.release() + self.cap = None +``` + +Reconnecting is not instantaneous — the stream needs its parameter sets again, so the wait described above repeats on every reopen. Reopening once per viewer session is inexpensive; reopening once per second is not. Keep the capture open while the view is open, and close it when the view closes. + +### For the ROS 2 Cameras, Destroy the Subscription + +The same principle applies to the RealSense Topics, and the mechanism is worth stating explicitly because the obvious approach does not work. + +**`rclpy` deserializes a message before the callback runs.** A callback that returns early on an unwanted frame has already paid for that frame in full. Three 640×480 color streams at 30 Hz is roughly 83 MB/s of pixel data copied into Python for images nobody is looking at, and an early `return` does not avoid any of it. The only way not to pay is not to be subscribed: + +```python +def set_camera_wanted(self, topic: str, wanted: bool) -> None: + """Create and destroy the subscription. Filtering in the callback is too late.""" + if wanted and topic not in self.subs: + self.subs[topic] = self.node.create_subscription( + Image, topic, self._on_image, qos) + elif not wanted and topic in self.subs: + self.node.destroy_subscription(self.subs.pop(topic)) +``` + +The same measurement on the point cloud, where the messages are larger, makes the size of the effect clear: a `/LIDAR/POINTS` reader held open while its view was watched cost 4.0% of one core doing nothing but deserializing and discarding 4.65 MB sweeps at approximately 8 Hz. Created, used for one sweep, and destroyed again, the same measurement was 0.1%. Endpoint discovery takes roughly 200 ms, which is what makes creating and destroying a subscription affordable at that rate. + +### Measured Result + +| Configuration | CPU | +| --- | --- | +| All four camera views open; only the encode skipped when unwatched | 121% of one core | +| Encode skipped, captures and subscriptions still held open | 91% of one core | +| Captures released and subscriptions destroyed; no view open | **3.26% of one core** | +| Captures released and subscriptions destroyed; one view open | **6.72% of one core** | + +The last two rows are the working configuration. Percentages are of a single core, as reported by `top` on the robot, and were observed during the session rather than recorded to a file; treat them as the size of the effect rather than as precise figures, and re-measure on your own unit. + +> [!TIP] +> +> A convenient way to hold this property is to make it a test rather than a habit. Reading the source back and failing if a capture is opened outside the watched path — or if a subscription is created unconditionally — costs very little and survives a hurried afternoon in a way that a comment does not. + +## A Minimal Reader + +The following program connects, waits for a decodable frame, prints its statistics, writes one JPEG, and releases the capture. It uses only OpenCV and is intended to be run on a development host on the robot network, or on the AOS host itself. + +```python +#!/usr/bin/env python3 +"""Read one usable frame from the DR02 Pro forward RGB camera.""" + +import os +import sys +import time + +# Must be set before the capture is opened. Setting it before importing cv2 is +# the reliable order. UDP is the default and loses fragments while the decoder +# waits for its H.265 parameter sets. +os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "rtsp_transport;tcp|max_delay;5000000" + +import cv2 + +URL = "rtsp://10.21.33.103:8554/video1" +MIN_STD = 10.0 # measured: 1.06 before decode, approximately 68 after +TIMEOUT_S = 20.0 + +cap = cv2.VideoCapture(URL, cv2.CAP_FFMPEG) +if not cap.isOpened(): + sys.exit(f"cannot open {URL}") + +deadline = time.monotonic() + TIMEOUT_S +frames = 0 +try: + while time.monotonic() < deadline: + ok, frame = cap.read() + if not ok: + continue + frames += 1 + gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) + std = float(gray.std()) + if std < MIN_STD: + # read() succeeded and the frame is still uniform grey. This is the + # decoder waiting for VPS/SPS/PPS, not a covered lens. + continue + focus = float(cv2.Laplacian(gray, cv2.CV_64F).var()) + print(f"frame {frames}: {frame.shape[1]}x{frame.shape[0]} " + f"std={std:.2f} focus={focus:.1f}") + cv2.imwrite("forward.jpg", frame) + break + else: + sys.exit(f"no decodable frame within {TIMEOUT_S:.0f}s " + f"({frames} frames read, all below std {MIN_STD})") +finally: + # Releasing the capture is what stops the H.265 decode. + cap.release() +``` + +Requires `python3-opencv` (or `pip install opencv-python`) on the machine running it. It is deliberately not built as a `dr02_pro` example, because that would add an OpenCV dependency to a package whose examples currently need only `rclcpp` and `drdds`. + +## Lens Characteristics + +The forward camera's lens is strongly wide-angle. A physically straight ceiling beam renders as a pronounced arc across the frame. + +This matters for fiducial markers. Marker detection fits straight-edged quadrilaterals, and barrel distortion bows the edges of a marker enough to defeat the quad test while the marker remains perfectly legible to a person. A marker that a human observer can read in the image may still not be detected. Undistort before detection, or place markers near the optical center where the distortion is smallest. + +The camera publishes no intrinsics. Any geometry derived from this image — an angle, a bearing, an artificial horizon — rests on an assumed focal length until the camera is calibrated. Label such values as estimated in the image itself, not only in the surrounding documentation, because a screenshot travels without its caption. + +## Troubleshooting + +### The Image Is a Uniform Grey Rectangle + +This is the decoder waiting for the H.265 parameter sets, not a covered lens and not a fault. Confirm by checking the frame's standard deviation: a value near 1 with a successful `read()` is the undecoded case. Force TCP transport and continue reading until the variance rises. See [Judge a Frame by Its Variance](#2-judge-a-frame-by-its-variance-not-by-whether-the-read-succeeded). + +### The Stream Connects and Then Stops + +Check for a second consumer. The server appeared to serve one client at a time, and a second connection evicts the first with `454 Session Not Found`. A forgotten `ffplay` or VLC window on a development host is enough. + +### The Stream Never Opens + +Confirm the path is `video1`. `video0` does not open on this robot. Confirm network reachability to the AOS host, and that the port is answering: + +```bash +ping -c 3 10.21.33.103 +ffprobe -rtsp_transport tcp -v error -show_streams rtsp://10.21.33.103:8554/video1 +``` + +### `/dev/video*` Reports Permission Denied + +The forward camera is also exposed as `/dev/video0` and `/dev/video1` on the AOS host. On the robot measured, the device nodes were `root:video` with mode `0660`, while the login account belonged to `sudo`, `adm`, and `audio` but **not** to `video`. The camera therefore appears broken rather than unreadable — an application reporting that it cannot open the device is reporting a group membership problem, not a hardware one. + +Prefer the RTSP stream, which needs no special group membership. If the device nodes are genuinely required, add the account to the `video` group: + +```bash +sudo usermod -aG video "$USER" +# Log out and back in for the new group to take effect. +id -nG +``` + +### CPU on the Robot Is High While a Camera View Is Open + +Confirm that unwatched cameras are actually released rather than read and discarded, and that ROS 2 camera subscriptions are destroyed rather than filtered in the callback. See [Keeping the Cost Off the Robot](#keeping-the-cost-off-the-robot). Reading and discarding pays the full decode and deserialization cost of every frame. + +## Related Documentation + +- [RealSense Cameras](CAMERA.md): the three USB depth cameras on the NOS host, their ROS 2 driver, and the librealsense interfaces. +- [Real-Robot Deployment and Control](REAL_ROBOT.md): host addresses and network access. +- [Developer Modes](DEVELOPER_MODE.md): mode switching. The relationship between Developer Mode and the RTSP service was not established during the measurements above; the stream was read without changing modes, but this was not tested systematically. diff --git a/src/dr02_pro/docs/FORWARD_CAMERA_CN.md b/src/dr02_pro/docs/FORWARD_CAMERA_CN.md new file mode 100644 index 0000000..818dba3 --- /dev/null +++ b/src/dr02_pro/docs/FORWARD_CAMERA_CN.md @@ -0,0 +1,285 @@ +# DR02 Pro 前向 RGB 相机使用指南 + +[返回 DR02 Pro SDK 使用指南](../README_CN.md) + +[RealSense 相机](CAMERA_CN.md)文档中介绍的三台 Intel RealSense D435 相机通过 USB 连接至 NOS 主机（`10.21.33.106`），并将数据发布为 ROS 2 Topic。在本文测量所使用的那台机器人上，这三台相机均向下倾斜安装，没有任何一台提供沿行进方向的前向视野。 + +该机器人的前向视野来自另一台独立的 RGB 相机，该相机挂在 AOS 主机（`10.21.33.103`）上。它不是 RealSense 相机，不由 `realsense2_camera` 打开，**也不发布任何 ROS 2 Topic**，只能通过 RTSP 视频流访问。本文说明如何读取该视频流、如何区分已解码图像与未解码图像，以及如何避免该视频流成为机器人上占用 CPU 最多的负载。 + +> [!NOTE] +> +> 本文所有内容均在 2026 年 9 月 3 日于一台 DR02 Pro 上实测得到——一台机器人，一天时间。主机地址、视频流路径和相机安装角度均为该机器人的实际情况。本文描述的各类现象的*形态*预计具有普遍性，但具体数值不作保证。在依赖这些信息之前，请先在自己的设备上确认相机安装角度和视频流地址。 + +## 相机访问方式 + +| 相机 | 主机 | 传输方式 | 访问方式 | +| --- | --- | --- | --- | +| 3 × RealSense D435 | NOS 主机（`10.21.33.106`） | USB | `realsense2_camera` ROS 2 驱动，或 librealsense SDK。参阅 [RealSense 相机](CAMERA_CN.md)。 | +| 前向 RGB | AOS 主机（`10.21.33.103`） | 经机器人网络的 RTSP | 任意 RTSP 客户端。不发布 ROS 2 Topic。 | + +前向相机的数据链路如下： + +```text +相机 -> AOS 主机上的 RTSP 服务 -> RTSP 客户端（OpenCV、FFmpeg、VLC、GStreamer） +``` + +如果其他 ROS 2 节点需要前向图像，应由用户程序读取 RTSP 流并自行发布所需消息。机器人上没有任何程序会自动完成这一步。 + +## 视频流地址 + +| 路径 | 结果 | +| --- | --- | +| `rtsp://10.21.33.103:8554/video1` | 提供 1280×720 的前向 RGB 视频流。 | +| `rtsp://10.21.33.103:8554/video0` | 无法打开。 | + +在编写任何代码之前，先用命令行确认视频流： + +```bash +# 在机器人网络中的开发主机上执行，或直接在 AOS 主机上执行 +ffprobe -rtsp_transport tcp -v error -show_streams \ + rtsp://10.21.33.103:8554/video1 +``` + +输出中报告的编码格式为 `hevc`，即 H.265。下一节中的大部分现象都源自这一点。 + +## 读取视频流 + +### 1. 必须在打开视频流之前强制使用 TCP + +OpenCV 的 FFmpeg 后端默认对 RTSP 使用 UDP 传输。在该视频流上，UDP 会在解码器尚不能输出图像的那段时间内丢失分片，并显著延长这段时间。应通过 FFmpeg 后端读取的环境变量设置传输方式。该变量必须在**打开视频流之前**设置；在 `import cv2` 之前设置是可靠的顺序： + +```python +import os + +os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "rtsp_transport;tcp|max_delay;5000000" + +import cv2 + +cap = cv2.VideoCapture("rtsp://10.21.33.103:8554/video1", cv2.CAP_FFMPEG) +``` + +直接使用 `ffmpeg` 或 `ffprobe` 时，对应参数是 `-rtsp_transport tcp`；使用 VLC 时是 `--rtsp-tcp`。 + +### 2. 应根据图像方差判断，而不是根据读取是否成功判断 + +> [!IMPORTANT] +> +> 使用默认参数打开时，该视频流能够正常连接，也能正确报告 1280×720 的分辨率，但返回的是**一致的灰色图像**，且 `cap.read()` 对其中每一帧都返回 `True`。相机返回的一致灰色图像看起来与镜头被遮挡完全一样，在产生本文的那次实测中，它最初正是被误判为镜头遮挡。实际上相机自始至终工作正常。 + +原因可以从 VLC 自身在同一视频流上的日志中看到，日志反复输出： + +```text +hevc packetizer: Waiting for VPS/SPS/PPS +``` + +该视频流为 H.265/HEVC 编码，其参数集（VPS、SPS、PPS）的发送间隔较长，并不会在每个关键帧之前发送。在参数集到达之前，任何解码器都无法输出图像。所有客户端在这段时间内显示的都是灰色，包括 VLC——它只是用黑色窗口把这段时间遮住，直到能够绘制画面为止。因此，取 `read()` 返回的第一帧的采集循环正好落在这段时间内；把读取成功当作图像有效的程序，会把一台工作正常的相机报告为故障相机。 + +图像自身的统计量可以清晰地区分这两种情况。在该视频流上的一次采集中实测： + +| 帧号 | 标准差 | 拉普拉斯方差 | 判读 | +| --- | --- | --- | --- | +| 0 | 1.06 | 6.2 | 一致灰色。`cap.read()` 返回 `True`。 | +| 20 及之后 | ≈68 | ≈330 | 正常图像。 | + +两组数值相差一个数量级以上，因此阈值无需精细调整： + +```python +import cv2 + + +def frame_is_decoded(frame, min_std: float = 10.0) -> bool: + """解码器收到参数集之后返回 True。 + + 在 VPS/SPS/PPS 到达之前产生的帧是一致灰色，而 read() 对其返回成功。 + 该视频流实测：之前标准差为 1.06，之后约为 68。取两者之间的任意阈值均可。 + """ + gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) + return float(gray.std()) >= min_std +``` + +拉普拉斯方差（`cv2.Laplacian(gray, cv2.CV_64F).var()`）同样能区分这两组数值，并可额外反映清晰度，但标准差计算量更小，单独使用已足以判断一帧是否完成解码。 + +在该检查通过之前应丢弃所有帧。在这台机器人上，图像大约在第 20 帧出现；建议设置一个较宽松的超时上限而不是固定帧数，并在超时后报告超时，而不是发布灰色图像。 + +### 3. 同一时间只能有一个客户端 + +该机器人上的 RTSP 服务表现为**同一时间只服务一个客户端**。在已有客户端连接的情况下打开第二个客户端，会导致第一个客户端被踢出并报错： + +```text +454 Session Not Found +``` + +应按单一读取端设计。如果多个使用方都需要前向图像，应只读取一次视频流再转发——发布为 ROS 2 Topic，或使用应用已有的传输方式——而不是让每个使用方都去连接同一个 RTSP 地址。请注意，开发主机上遗留的 `ffplay`、`ffprobe` 或 VLC 窗口同样是一个客户端，这也是应用程序的视频流莫名中断的常见原因。 + +## 不要把开销留给机器人 + +在 1280×720 分辨率下解码 H.265 并非没有代价，而这部分 CPU 属于机器人。在产生本文的那次实测中，一个同时显示四路相机的监控页面成为了机器人上占用 CPU 最多的单个负载——超过了激光雷达驱动——达到**一个核的 91%**。查看相机画面的操作员，不应该与机器人自身的感知程序争抢它正站立其上的这台机器的算力。 + +有两项改动解决了这个问题，二者遵循同一个原则：**不要为没有人在看的图像付出代价。** + +### 应释放采集句柄，而不是读取后丢弃 + +读取图像再丢弃，仍然会对每一帧完整解码。解码才是主要开销，之后的编码和显示相对次之。仅跳过编码而继续读取，只把进程从一个核的 121% 降到 91%，并不够。 + +真正消除开销的做法是关闭采集： + +```python +class ForwardCamera: + """只在有人查看时保持 RTSP 采集处于打开状态。""" + + URL = "rtsp://10.21.33.103:8554/video1" + + def __init__(self): + self.cap = None + + def open(self): + if self.cap is None: + self.cap = cv2.VideoCapture(self.URL, cv2.CAP_FFMPEG) + return self.cap.isOpened() + + def close(self): + # 不是「停止编码」，也不是「读取后丢弃」。H.265 解码才是开销所在， + # 只有释放采集句柄才能真正停止它。 + if self.cap is not None: + self.cap.release() + self.cap = None +``` + +重新连接并非瞬时完成——视频流需要重新等待参数集，因此上文描述的等待过程会在每次重新打开时重复一遍。每个查看会话重新打开一次的代价很小，每秒重新打开一次则不然。应在视图打开期间保持采集，视图关闭时再关闭采集。 + +### 对于 ROS 2 相机，应销毁订阅 + +同一原则也适用于 RealSense 的 Topic，而且这里的机制值得明确说明，因为看起来最直接的做法并不奏效。 + +**`rclpy` 会在回调执行之前完成消息反序列化。** 对不需要的图像在回调中提前 `return`，此时代价已经完整付出。三路 640×480 彩色图像以 30 Hz 发布，约相当于每秒 83 MB 的像素数据被复制进 Python，而这些图像根本没有人在看，提前 `return` 无法省下其中任何一部分。唯一不付出代价的方法就是不订阅： + +```python +def set_camera_wanted(self, topic: str, wanted: bool) -> None: + """创建和销毁订阅。在回调中过滤为时已晚。""" + if wanted and topic not in self.subs: + self.subs[topic] = self.node.create_subscription( + Image, topic, self._on_image, qos) + elif not wanted and topic in self.subs: + self.node.destroy_subscription(self.subs.pop(topic)) +``` + +在消息更大的点云上进行同样的测量，可以更清楚地看出这一效应的量级：在视图被查看期间一直保持打开的 `/LIDAR/POINTS` 订阅，仅仅是反序列化并丢弃约 8 Hz、每帧 4.65 MB 的点云，就占用了一个核的 4.0%；改为每次只为一帧点云创建订阅、用完即销毁后，同样的测量结果是 0.1%。端点发现约需 200 ms，这正是使得按此频率创建和销毁订阅仍然划算的原因。 + +### 实测结果 + +| 配置 | CPU | +| --- | --- | +| 四路相机视图全开；无人查看时仅跳过编码 | 一个核的 121% | +| 跳过编码，但采集句柄与订阅仍保持打开 | 一个核的 91% | +| 释放采集句柄并销毁订阅；未打开任何视图 | **一个核的 3.26%** | +| 释放采集句柄并销毁订阅；打开一路视图 | **一个核的 6.72%** | + +最后两行是最终采用的配置。百分比均以单核为基准，取自机器人上 `top` 的输出，是实测过程中的观察值而非记录到文件的数据；请将其视为该效应的量级，而不是精确数值，并在自己的设备上重新测量。 + +> [!TIP] +> +> 保持这一特性的一个便捷做法是把它写成测试，而不是依靠习惯。回读源码，一旦发现在「被查看」路径之外打开了采集句柄，或者无条件创建了订阅就让测试失败——这样做代价很小，并且能在忙乱的下午里守住这一约束，而注释做不到这一点。 + +## 最小读取程序 + +以下程序完成连接、等待一帧可用图像、打印其统计量、写出一张 JPEG，然后释放采集句柄。它只依赖 OpenCV，可在机器人网络中的开发主机上运行，也可直接在 AOS 主机上运行。 + +```python +#!/usr/bin/env python3 +"""从 DR02 Pro 前向 RGB 相机读取一帧可用图像。""" + +import os +import sys +import time + +# 必须在打开采集之前设置。在 import cv2 之前设置是可靠的顺序。 +# 默认传输方式为 UDP，会在解码器等待 H.265 参数集期间丢失分片。 +os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "rtsp_transport;tcp|max_delay;5000000" + +import cv2 + +URL = "rtsp://10.21.33.103:8554/video1" +MIN_STD = 10.0 # 实测：解码前 1.06，解码后约 68 +TIMEOUT_S = 20.0 + +cap = cv2.VideoCapture(URL, cv2.CAP_FFMPEG) +if not cap.isOpened(): + sys.exit(f"cannot open {URL}") + +deadline = time.monotonic() + TIMEOUT_S +frames = 0 +try: + while time.monotonic() < deadline: + ok, frame = cap.read() + if not ok: + continue + frames += 1 + gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) + std = float(gray.std()) + if std < MIN_STD: + # read() 成功，但图像仍是一致灰色。这是解码器在等待 + # VPS/SPS/PPS，而不是镜头被遮挡。 + continue + focus = float(cv2.Laplacian(gray, cv2.CV_64F).var()) + print(f"frame {frames}: {frame.shape[1]}x{frame.shape[0]} " + f"std={std:.2f} focus={focus:.1f}") + cv2.imwrite("forward.jpg", frame) + break + else: + sys.exit(f"no decodable frame within {TIMEOUT_S:.0f}s " + f"({frames} frames read, all below std {MIN_STD})") +finally: + # 释放采集句柄才能真正停止 H.265 解码。 + cap.release() +``` + +运行该程序的机器需要安装 `python3-opencv`（或执行 `pip install opencv-python`）。本文有意没有把它做成 `dr02_pro` 的编译示例，因为那样会给一个当前仅依赖 `rclcpp` 和 `drdds` 的软件包引入 OpenCV 依赖。 + +## 镜头特性 + +前向相机的镜头广角很强。物理上笔直的天花板横梁在画面中会呈现为明显的弧线。 + +这一点对视觉标记（fiducial marker）有影响。标记检测依赖对直边四边形的拟合，而桶形畸变会把标记的边缘弯曲到足以让四边形检测失败的程度，但此时标记对人眼仍然完全清晰可读。也就是说，人能在图像中读出的标记，程序未必能检测到。建议在检测之前先做去畸变，或者把标记放在畸变最小的光心附近。 + +该相机不发布任何内参。凡是由该图像推导出的几何量——角度、方位、人工地平线——在完成标定之前都建立在假定的焦距之上。这类数值应在图像本身上标注为估计值，而不只是写在配套文档里，因为截图在传播时不会带着它的说明文字。 + +## 常见问题 + +### 图像是一整块灰色 + +这是解码器在等待 H.265 参数集，既不是镜头被遮挡，也不是故障。可通过图像标准差确认：`read()` 成功而标准差接近 1，即为未解码的情况。请强制使用 TCP 传输，并继续读取直到方差上升。参阅本文「应根据图像方差判断，而不是根据读取是否成功判断」一节。 + +### 视频流连接后中断 + +请检查是否存在第二个客户端。该服务表现为同一时间只服务一个客户端，第二个连接会把第一个踢出并报 `454 Session Not Found`。开发主机上被遗忘的 `ffplay` 或 VLC 窗口就足以造成这一现象。 + +### 视频流始终无法打开 + +确认路径为 `video1`。在这台机器人上 `video0` 无法打开。确认到 AOS 主机的网络可达，并确认端口有响应： + +```bash +ping -c 3 10.21.33.103 +ffprobe -rtsp_transport tcp -v error -show_streams rtsp://10.21.33.103:8554/video1 +``` + +### `/dev/video*` 提示权限不足 + +前向相机在 AOS 主机上同时以 `/dev/video0` 和 `/dev/video1` 暴露。在实测的这台机器人上，设备节点属主为 `root:video`、权限为 `0660`，而登录账号属于 `sudo`、`adm` 和 `audio` 组，但**不**属于 `video` 组。因此这台相机看起来像是坏了，而不是不可读——程序报告无法打开设备，实际反映的是用户组权限问题，而不是硬件问题。 + +建议优先使用 RTSP 流，它不需要任何特殊的用户组权限。如果确实需要使用设备节点，请把账号加入 `video` 组： + +```bash +sudo usermod -aG video "$USER" +# 需要重新登录，新的用户组才会生效。 +id -nG +``` + +### 打开相机视图时机器人 CPU 占用很高 + +确认无人查看的相机确实被释放，而不是读取后丢弃；确认 ROS 2 相机订阅确实被销毁，而不是在回调中过滤。参阅本文「不要把开销留给机器人」一节。读取后丢弃仍然要为每一帧付出完整的解码和反序列化代价。 + +## 相关文档 + +- [RealSense 相机](CAMERA_CN.md)：NOS 主机上的三台 USB 深度相机、ROS 2 驱动及 librealsense 接口。 +- [实机部署与控制](REAL_ROBOT_CN.md)：主机地址与网络访问方式。 +- [开发者模式](DEVELOPER_MODE_CN.md)：模式切换。上述实测过程中没有确定开发者模式与 RTSP 服务之间的关系；当时未切换模式即可读取视频流，但这一点未经系统性验证。