BEVFusion

本仓库说明：nuScenes mini 目标检测和语义分割复现

本仓库是在 MIT Han Lab 官方 BEVFusion 代码基础上整理的复现版本，当前已完成 nuScenes mini 目标检测 和 nuScenes mini BEV map segmentation。官方原始仓库为：

https://github.com/mit-han-lab/bevfusion

本次复现基于官方提交：

326653dc06e0938edf1aae7d01efcd158ba83de5

当前仓库额外包含：

• 适配当前机器和 CUDA/PyTorch 环境的代码修复。
• nuScenes mini 目标检测评测脚本和数据准备脚本。
• mini val 目标检测评测记录、类别指标记录、可视化生成指令。
• 本地复现生成的目标检测可视化 GIF 和图片样例。
• nuScenes mini BEV map segmentation 评测记录、可视化生成指令和预测/GT 对比 GIF。
• 原始文件来源说明和感谢说明。

详细复现说明见：

docs/目标检测复现说明.md
docs/语义分割复现说明.md
docs/复现进度.md
docs/具体操作记录.md
docs/原始文件来源说明.md

mini val 目标检测结果

本次使用官方 BEVFusion camera+lidar detection checkpoint 在 nuScenes mini val 上完成目标检测评测。mini 数据集很小，结果只用于复现流程验证，不代表完整 nuScenes validation/test 排名。

指标	结果
mAP	0.5732
NDS	0.5800

各类别 AP：

类别	AP
car	0.916
truck	0.817
bus	0.995
trailer	0.000
construction_vehicle	0.000
pedestrian	0.919
motorcycle	0.703
bicycle	0.518
traffic_cone	0.865
barrier	0.000

本次复现可视化结果

目标检测 + BEV map segmentation 对齐动图：

相机 + LiDAR BEV 综合动图：

LiDAR BEV 动图：

样例帧：

camera 0	camera 1	camera 2

camera 3	camera 4	camera 5

LiDAR BEV

mini val BEV map segmentation 结果

本次使用官方 BEVFusion camera+lidar segmentation checkpoint 在 nuScenes mini val 上完成 BEV map segmentation 评测。该任务输出 BEV 地图语义类别，不是相机图像逐像素分割，也不是点云逐点分割；配置中 use_radar: false，没有使用 radar。

指标	结果
map/mean/iou@max	0.7184
drivable_area	0.8931
ped_crossing	0.8050
walkway	0.7245
stop_line	0.5219
carpark_area	0.7756
divider	0.5906

预测 map 与 GT map 对比动图：

完整预测和 GT 可视化输出位于本地 outputs/mini_val_seg_pred_vis/、outputs/mini_val_seg_gt_vis/ 和 outputs/mini_val_seg_compare_index.html，目录体积较大未提交到 GitHub；GitHub 中提交 GIF 和一张 PNG 样例。说明见：

mini_val_seg_vis_README.md

来源与感谢

感谢 MIT Han Lab 开源 BEVFusion 项目和论文代码，感谢 nuScenes 提供数据集，感谢 OpenMMLab/MMDetection3D、MMCV、PyTorch 等开源项目提供基础工具。本仓库保留原始 LICENSE，并在 docs/原始文件来源说明.md 中记录了官方源码来源、基准提交、本次新增/修改内容，以及未上传的大文件来源。

下面保留官方 BEVFusion 原始 README 内容，便于对照官方说明。

website | paper | video

本仓库复现生成的目标检测和 BEV map segmentation 动图：

News

• (2024/5) BEVFusion is integrated into NVIDIA DeepStream for sensor fusion.
• (2023/5) NVIDIA provides a TensorRT deployment solution of BEVFusion, achieving 25 FPS on Jetson Orin.
• (2023/4) BEVFusion ranks first on Argoverse 3D object detection leaderboard among all solutions.
• (2023/1) BEVFusion is integrated into MMDetection3D.
• (2023/1) BEVFusion is accepted to ICRA 2023!
• (2022/8) BEVFusion ranks first on Waymo 3D object detection leaderboard among all solutions.
• (2022/6) BEVFusion ranks first on nuScenes 3D object detection leaderboard among all solutions.
• (2022/6) BEVFusion ranks first on nuScenes 3D object detection leaderboard among all solutions.

Abstract

Multi-sensor fusion is essential for an accurate and reliable autonomous driving system. Recent approaches are based on point-level fusion: augmenting the LiDAR point cloud with camera features. However, the camera-to-LiDAR projection throws away the semantic density of camera features, hindering the effectiveness of such methods, especially for semantic-oriented tasks (such as 3D scene segmentation). In this paper, we break this deeply-rooted convention with BEVFusion, an efficient and generic multi-task multi-sensor fusion framework. It unifies multi-modal features in the shared bird's-eye view (BEV) representation space, which nicely preserves both geometric and semantic information. To achieve this, we diagnose and lift key efficiency bottlenecks in the view transformation with optimized BEV pooling, reducing latency by more than 40x. BEVFusion is fundamentally task-agnostic and seamlessly supports different 3D perception tasks with almost no architectural changes. It establishes the new state of the art on the nuScenes benchmark, achieving 1.3% higher mAP and NDS on 3D object detection and 13.6% higher mIoU on BEV map segmentation, with 1.9x lower computation cost.

Results

3D Object Detection (on Waymo test)

Model	mAP-L1	mAPH-L1	mAP-L2	mAPH-L2
BEVFusion	82.72	81.35	77.65	76.33
BEVFusion-TTA	86.04	84.76	81.22	79.97

Here, BEVFusion only uses a single model without any test time augmentation. BEVFusion-TTA uses single model with test-time augmentation and no model ensembling is applied.

3D Object Detection (on nuScenes test)

Model	Modality	mAP	NDS
BEVFusion-e	C+L	74.99	76.09
BEVFusion	C+L	70.23	72.88
BEVFusion-base*	C+L	71.72	73.83

*: We scaled up MACs of the model to match the computation cost of concurrent work.

3D Object Detection (on nuScenes validation)

Model	Modality	mAP	NDS	Checkpoint
BEVFusion	C+L	68.52	71.38	Link
Camera-Only Baseline	C	35.56	41.21	Link
LiDAR-Only Baseline	L	64.68	69.28	Link

Note: The camera-only object detection baseline is a variant of BEVDet-Tiny with a much heavier view transformer and other differences in hyperparameters. Thanks to our efficient BEV pooling operator, this model runs fast and has higher mAP than BEVDet-Tiny under the same input resolution. Please refer to BEVDet repo for the original BEVDet-Tiny implementation. The LiDAR-only baseline is TransFusion-L.

BEV Map Segmentation (on nuScenes validation)

Model	Modality	mIoU	Checkpoint
BEVFusion	C+L	62.95	Link
Camera-Only Baseline	C	57.09	Link
LiDAR-Only Baseline	L	48.56	Link

Usage

Prerequisites

The code is built with following libraries:

• Python >= 3.8, <3.9
• OpenMPI = 4.0.4 and mpi4py = 3.0.3 (Needed for torchpack)
• Pillow = 8.4.0 (see here)
• PyTorch >= 1.9, <= 1.10.2
• tqdm
• torchpack
• mmcv = 1.4.0
• mmdetection = 2.20.0
• nuscenes-dev-kit

After installing these dependencies, please run this command to install the codebase:

python setup.py develop

We also provide a Dockerfile to ease environment setup. To get started with docker, please make sure that nvidia-docker is installed on your machine. After that, please execute the following command to build the docker image:

cd docker && docker build . -t bevfusion

We can then run the docker with the following command:

nvidia-docker run -it -v `pwd`/../data:/dataset --shm-size 16g bevfusion /bin/bash

We recommend the users to run data preparation (instructions are available in the next section) outside the docker if possible. Note that the dataset directory should be an absolute path. Within the docker, please run the following command to clone our repo and install custom CUDA extensions:

cd home && git clone https://github.com/mit-han-lab/bevfusion && cd bevfusion
python setup.py develop

You can then create a symbolic link data to the /dataset directory in the docker.

Data Preparation

nuScenes

Please follow the instructions from here to download and preprocess the nuScenes dataset. Please remember to download both detection dataset and the map extension (for BEV map segmentation). After data preparation, you will be able to see the following directory structure (as is indicated in mmdetection3d):

mmdetection3d
├── mmdet3d
├── tools
├── configs
├── data
│   ├── nuscenes
│   │   ├── maps
│   │   ├── samples
│   │   ├── sweeps
│   │   ├── v1.0-test
|   |   ├── v1.0-trainval
│   │   ├── nuscenes_database
│   │   ├── nuscenes_infos_train.pkl
│   │   ├── nuscenes_infos_val.pkl
│   │   ├── nuscenes_infos_test.pkl
│   │   ├── nuscenes_dbinfos_train.pkl

Evaluation

We also provide instructions for evaluating our pretrained models. Please download the checkpoints using the following script:

./tools/download_pretrained.sh

Then, you will be able to run:

torchpack dist-run -np [number of gpus] python tools/test.py [config file path] pretrained/[checkpoint name].pth --eval [evaluation type]

For example, if you want to evaluate the detection variant of BEVFusion, you can try:

torchpack dist-run -np 8 python tools/test.py configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml pretrained/bevfusion-det.pth --eval bbox

While for the segmentation variant of BEVFusion, this command will be helpful:

torchpack dist-run -np 8 python tools/test.py configs/nuscenes/seg/fusion-bev256d2-lss.yaml pretrained/bevfusion-seg.pth --eval map

Training

We provide instructions to reproduce our results on nuScenes.

For example, if you want to train the camera-only variant for object detection, please run:

torchpack dist-run -np 8 python tools/train.py configs/nuscenes/det/centerhead/lssfpn/camera/256x704/swint/default.yaml --model.encoders.camera.backbone.init_cfg.checkpoint pretrained/swint-nuimages-pretrained.pth

For camera-only BEV segmentation model, please run:

torchpack dist-run -np 8 python tools/train.py configs/nuscenes/seg/camera-bev256d2.yaml --model.encoders.camera.backbone.init_cfg.checkpoint pretrained/swint-nuimages-pretrained.pth

For LiDAR-only detector, please run:

torchpack dist-run -np 8 python tools/train.py configs/nuscenes/det/transfusion/secfpn/lidar/voxelnet_0p075.yaml

For LiDAR-only BEV segmentation model, please run:

torchpack dist-run -np 8 python tools/train.py configs/nuscenes/seg/lidar-centerpoint-bev128.yaml

For BEVFusion detection model, please run:

torchpack dist-run -np 8 python tools/train.py configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml --model.encoders.camera.backbone.init_cfg.checkpoint pretrained/swint-nuimages-pretrained.pth --load_from pretrained/lidar-only-det.pth 

For BEVFusion segmentation model, please run:

torchpack dist-run -np 8 python tools/train.py configs/nuscenes/seg/fusion-bev256d2-lss.yaml --model.encoders.camera.backbone.init_cfg.checkpoint pretrained/swint-nuimages-pretrained.pth

Note: please run tools/test.py separately after training to get the final evaluation metrics.

Deployment on TensorRT

CUDA-BEVFusion: Best practice for TensorRT, which provides INT8 acceleration solutions and achieves 25fps on ORIN.

FAQs

Q: Can we directly use the info files prepared by mmdetection3d?

A: We recommend re-generating the info files using this codebase since we forked mmdetection3d before their coordinate system refactoring.

Acknowledgements

BEVFusion is based on mmdetection3d. It is also greatly inspired by the following outstanding contributions to the open-source community: LSS, BEVDet, TransFusion, CenterPoint, MVP, FUTR3D, CVT and DETR3D.

Please also check out related papers in the camera-only 3D perception community such as BEVDet4D, BEVerse, BEVFormer, M2BEV, PETR and PETRv2, which might be interesting future extensions to BEVFusion.

Citation

If BEVFusion is useful or relevant to your research, please kindly recognize our contributions by citing our paper:

@inproceedings{liu2022bevfusion,
  title={BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation},
  author={Liu, Zhijian and Tang, Haotian and Amini, Alexander and Yang, Xingyu and Mao, Huizi and Rus, Daniela and Han, Song},
  booktitle={IEEE International Conference on Robotics and Automation (ICRA)},
  year={2023}
}