Memory usage issues

[PIZHIPAI]

I encountered the following problem when running my data.
CUDA out of memory. Tried to allocate 43.34 GiB. GPU File "/root/pycharm/skyfall-gs-main_dmz/gaussian_renderer/init.py", line 135, in render rendered_image, rendered_depth, rendered_norm, rendered_alpha, radii, extra = rasterizer( File "/root/pycharm/skyfall-gs-main_dmz/train.py", line 196, in training render_pkg = render( File "/root/pycharm/skyfall-gs-main_dmz/train.py", line 1144, in training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 43.34 GiB. GPU

I'd like to ask if the issue is with diff-gaussian-rasterization-depth or something else? I've turned off pseudo-projection constraints and MLP, but I'm still experiencing memory issues. Running other 3DGS code doesn't present this problem. Do you know where the problem lies and why so much memory is being needed, even after only iterating to around 5000 iterations?

[jayin92]

Hi @PIZHIPAI ,

Sorry for the late reply. May I ask which stages are you training? If you are training the second stage, then FLUX.1 requires a significant amount of memory, which might cause OOM issue.

Best,
JIe-Ying