Approximately how many GPU resources are required to perform ASVR training?

[SensenGao]

[AlenjandroWang]

We utilize 4–8× A100 (80GB) GPUs
For training on fewer GPUs, reduce the per_device_train_batch_size and increase the gradient_accumulation_steps accordingly, ensuring that the global batch size remains constant:
per_device_train_batch_size × gradient_accumulation_steps × num_gpus.
For the pretraining stage, maintain a global batch size of 256.
For the instruction tuning stage, maintain a global batch size of 128.