跑训练代码最低配置是什么？

[huangshaoguang]

您好博主！请问一下跑训练代码最低配置是多少？本人尝试在单个RTX4070显卡上，num-envs =4和buffer-size=512都显存爆了，而且每次跑卡几个小时才报这个显存爆了的错误

[Simease]

我32g的4080 用num-envs =1024和buffer-size=2048000这个配置差不多是27G

[huangshaoguang]

我32g的4080 用num-envs =1024和buffer-size=2048000这个配置差不多是27G

我的只有12G

[huangshaoguang]

我32g的4080 用num-envs =1024和buffer-size=2048000这个配置差不多是27G

我不明白为什么跑了那么久才给我报OOM错误

[huangshaoguang]

我换了A10（24G）和4090的显卡，一直卡在输出Starting evalutaion at time 0这里，起码三四个小时，完后还是卡在这里又是好多个小时：/mnt/workspace/UFO/.venv/lib/python3.10/site-packages/torch/autograd/graph.py:824: UserWarning: Attempting to run cuBLAS, but there was no current CUDA context! Attempting to set the primary context... (Triggered internally at /pytorch/aten/src/ATen/cuda/CublasHandlePool.cpp:181.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
W0903 15:12:08.359000 3616 .venv/lib/python3.10/site-packages/torch/_logging/_internal.py:1130] [7/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
/mnt/workspace/UFO/.venv/lib/python3.10/site-packages/torch/_inductor/lowering.py:1725: FutureWarning: torch._prims_common.check is deprecated and will be removed in the future. Please use torch._check* functions instead.
check(
/mnt/workspace/UFO/.venv/lib/python3.10/site-packages/torch/_inductor/lowering.py:1725: FutureWarning: torch._prims_common.check is deprecated and will be removed in the future. Please use torch._check* functions instead.
check(

[huangshaoguang]

为了跑./run_train.sh --agent fb --robot-config configs/robots/g1_29dof.yaml --data-manifest configs/data/my_motion_auto_build.yaml --gpu-ids single --num-envs 1 --buffer-size 1024 --work-dir /tmp/ufo_smoke_my_robot
我换了A10（24G）和4090的显卡，一直卡在输出Starting evalutaion at time 0这里，起码三四个小时，完后还是卡在这里又是好多个小时：/mnt/workspace/UFO/.venv/lib/python3.10/site-packages/torch/autograd/graph.py:824: UserWarning: Attempting to run cuBLAS, but there was no current CUDA context! Attempting to set the primary context... (Triggered internally at /pytorch/aten/src/ATen/cuda/CublasHandlePool.cpp:181.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
W0903 15:12:08.359000 3616 .venv/lib/python3.10/site-packages/torch/_logging/_internal.py:1130] [7/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
/mnt/workspace/UFO/.venv/lib/python3.10/site-packages/torch/_inductor/lowering.py:1725: FutureWarning: torch._prims_common.check is deprecated and will be removed in the future. Please use torch._check* functions instead.
check(
/mnt/workspace/UFO/.venv/lib/python3.10/site-packages/torch/_inductor/lowering.py:1725: FutureWarning: torch._prims_common.check is deprecated and will be removed in the future. Please use torch._check* functions instead.
check(
真心请教一下up主该用啥配置电脑才能带的动？这个项目碰了好多坑了，搞了好久啊，不想放弃

[ShealyXie]

同问，我是RTXada4000 20G的显存，减少envs数目或者缓存大小都会出现同样的报错 T T
“UserWarning: Attempting to run cuBLAS, but there was no current CUDA context! Attempting to set the primary context... (Triggered internally at /pytorch/aten/src/ATen/cuda/CublasHandlePool.cpp:181.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
W0903 15:12:08.359000 3616 .venv/lib/python3.10/site-packages/torch/_logging/_internal.py:1130] [7/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
/mnt/workspace/UFO/.venv/lib/python3.10/site-packages/torch/_inductor/lowering.py:1725: FutureWarning: torch._prims_common.check is deprecated and will be removed in the future. Please use torch._check* functions instead.
check(
/mnt/workspace/UFO/.venv/lib/python3.10/site-packages/torch/_inductor/lowering.py:1725: FutureWarning: torch._prims_common.check is deprecated and will be removed in the future. Please use torch._check* functions instead.”

[huangshaoguang]

@ShealyXie 你这不是报错，是警告

[ShealyXie]

@ShealyXie 你这不是报错，是警告

我的也是一直卡在这里，留意到你最新的issue，请问这个问题是怎么解决的呢？直接换更高的配置吗😂？

[huangshaoguang]

@ShealyXie 我24G的显存，跑这个./run_train.sh --agent fb --robot-config configs/robots/g1_29dof.yaml --data-manifest configs/data/my_motion_auto_build.yaml --gpu-ids single --num-envs 512 --buffer-size 1024000 --work-dir /tmp/ufo_smoke_my_robot
没问题。
但是我看它提示占用了19G的显存，你20G的显存试试吧。它会跑半个小时左右才会有输出Q_fb，actor_loss等一大堆数据。

[Xuewang01]

你好，根据你提供的日志好像并没有报错，可以尝试在训练的时候把buffer放在cpu上降低显存占用（小于20gb
），我们在训练的时候最低配置使用了8卡4090（24gb）。另外，关于您提到的警告信息，这通常是CUDA上下文问题。