While training models in machine learning, why is it sometimes advantageous to keep the batch size to a power of 2? I thought it would be best to use a size that is the largest fit in your GPU memory / RAM.

This answer claims that for some packages, a power of 2 is better as a batch size. Can someone provide a detailed explanation / link to a detailed explanation for this? Is this true for all optimisation algorithms (gradient descent, backpropagation, etc) or only some of them?

up vote 8 down vote accepted

This is a problem of alignment of the virtual processors (VP) onto the physical processors (PP) of the GPU. Since the number of PP is often a power of 2, using a number of VP different from a power of 2 leads to poor performance.
You can see the mapping of the VP onto the PP as a pile of slices of size the number of PP.
Say you've got 16 PP.
You can map 16 VP on them : 1 VP is mapped onto 1 PP.
You can map 32 VP on them : 2 slices of 16 VP, 1 PP will be responsible for 2 VP.
Etc. During execution, each PP will execute the job of the 1st VP he is responsible for, then the job of the 2nd VP etc.
If you use 17 VP, each PP will execute the job of their 1st PP, then 1 PP will execute the job of the 17th AND the other ones will do nothing (precised below).
This is due to the SIMD paradigm (called vector in the 70s) used by GPUs. This is often called Data Parallelism : all the PP do the same thing at the same time but on different data. See https://en.wikipedia.org/wiki/SIMD.
More precisely, in the example with 17 VP, once the job of the 1st slice done (by all the PPs doing the job of their 1st VP), all the PP will do the same job (2nd VP), but only one has some data to work on.
Nothing to do with learning. This is only programming stuff.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.