Observations on Training Efficiency with Different Masking Strategies

[LuciferZap]

Thank you for your great work! I tried to reproduce the first two stages of training and conducted three experiments on A800 GPUs. Below are the details:

Experiment 1: Fully used 75% mask training for 227K steps, with a batch size of 2048.

Experiment 2:（github page recommended)Used 75% mask training for 179K steps, followed by 0% mask training for 48K steps, with a batch size of 2048.

Experiment 3: Used 0% mask training for 138K steps, with a batch size of 2048.

I found that Experiment 3 achieved the highest training efficiency. The computational cost across all three experiments was roughly the same. However, I did not observe any efficiency improvement from using mask training. Based on the final visualization results, the samples from Experiment 3 also appear to be the best.

If I missed any important details, please let me know!

Note on Resources: Due to instability in my resource pool, these experiments were sometimes run on 3 A800 GPUs and other times on a single GPU. However, the total computational cost remained consistent.

Attached below are the detailed visualization results:

[LuciferZap]

By the way, the dataset used to train the model are

1. Brushdata from Brushnet https://github.com/TencentARC/BrushNet?tab=readme-ov-file
2. sa1b with internvl caption https://huggingface.co/datasets/OpenGVLab/InternVL-SA-1B-Caption
3. JourneyDB dataset https://journeydb.github.io/

[xiankgx]

What do you mean when you say "method 3 achieved highest efficiency"? Do you mean the generated images look the greatest? Of course it should be. But masking means fewer tokens are used to train the diffusion model meaning faster training time which is probably the largest submodel to train in the composed model.

[LuciferZap]

我的意思是在同等算力投入的情况下，mask training并没有训练效率的提升，直接使用非mask的训练效果最好，按照文章的说法mask training应该是能加速收敛的，也就是最开始我使用75%的mask training做预训练再加上非mask的的SFT这个组合下收敛速度应该是最快的。但是我的实验做下来发现直接使用非mask从头训练模型的效果最好。

[LuciferZap]

这个现象跟MAE很像，虽然mask auto encoder的设计提升了训练的速度，但是需要3倍的epoch才能收敛。

[xiankgx]

How long does it takes for experiment #2 and #3, in terms of hours?

179k + 48k vs 138k.

Does it take less time to complete experiment #2 than #3?

[LuciferZap]

The time cost is almost same for exprimnet #1, #2 and #3, because the compute budge are same.