What's the advantage of multi-gpu training in real?

The decreasing speed of training loss is almost the same between one gpu and multi-gpu.

After averaging the gradients, the only benefit from multi-gpu is that the model seems to see more data in the same time.

But why average the gradients? Is it that the model is indeed feed with more data in the same time?

I see two main advantages of using multi-GPU instead of one as they distribute certain resources:

• using large DNN models - some recent models occupy vast space in memory so they simply cannot fit regular GPU and using multiple GPU allow to distribute some parts of the model to different GPU instances.
• speed-up DNN training is also a very positive effect of using multiple GPU but only if you have a high-speed connection among GPUs as NVIDIA came with their NVLink
• Oh, I make a mistake. The total gradient is still gradient_average * n_gpu. And each model on each GPU is applied by gradient_average, not total_gradient Dec 27 '18 at 9:42

Actually, with more GPUs you distribute the calculations and run them parallel. As an example, you can take the group concept used in AlexNet. Although, after employing that it was observed that it can have other properties but one of the main purposes of using SLI is due to the fact that you can distribute the group convolutions among multiple GPUs which can facilitate the convolution operations. Each update is done in the corresponding GPU.

• Oh, I make a mistake. The total gradient is still gradient_average * n_gpu. And each model on each GPU is applied by gradient_average, not total_gradient Dec 27 '18 at 9:42

More gpu means more data in a batch. And the gradients of a batch data is averaged for back-propagation.

If the learning rate of a batch is fixed, then the learning rate of a data is smaller.

If the learning rate of a data is fixed, then the learning rate of a batch is larger.

https://github.com/guotong1988/BERT-GPU

• This answer is based on my understanding now. Feb 5 at 4:40