While training models in machine learning, why is it sometimes advantageous to keep the batch size to a power of 2? I thought it would be best to use a size that is the largest fit in your GPU memory / RAM.

This answer claims that for some packages, a power of 2 is better as a batch size. Can someone provide a detailed explanation / link to a detailed explanation for this? Is this true for all optimisation algorithms (gradient descent, backpropagation, etc) or only some of them?


This is a problem of alignment of the virtual processors (VP) onto the physical processors (PP) of the GPU. Since the number of PP is often a power of 2, using a number of VP different from a power of 2 leads to poor performance.
You can see the mapping of the VP onto the PP as a pile of slices of size the number of PP.
Say you've got 16 PP.
You can map 16 VP on them : 1 VP is mapped onto 1 PP.
You can map 32 VP on them : 2 slices of 16 VP, 1 PP will be responsible for 2 VP.
Etc. During execution, each PP will execute the job of the 1st VP he is responsible for, then the job of the 2nd VP etc.
If you use 17 VP, each PP will execute the job of their 1st PP, then 1 PP will execute the job of the 17th AND the other ones will do nothing (precised below).
This is due to the SIMD paradigm (called vector in the 70s) used by GPUs. This is often called Data Parallelism : all the PP do the same thing at the same time but on different data. See https://en.wikipedia.org/wiki/SIMD.
More precisely, in the example with 17 VP, once the job of the 1st slice done (by all the PPs doing the job of their 1st VP), all the PP will do the same job (2nd VP), but only one has some data to work on.
Nothing to do with learning. This is only programming stuff.

| improve this answer | |
  • 1
    $\begingroup$ would it be more accurate to say that batch sizes should then be a multiple of the number of PP? That is, in your example we could map 16x3=48 VP to 16 PP? $\endgroup$ – 1west Feb 12 '19 at 21:47
  • $\begingroup$ Yes. well... If you do the mapping VP -> PP, yourself, sure you're 100% right. If you use a library, and ask for 80 VP. I'm not sure. I'm not saying you're wrong. If the ratio is a power of 2, you can use very classical and easy optimizations.Think of memory accesses. If the # of upstairs neighbours of a VP is not a power of 2, say 5, the lib won't be able to use classical O(log_2(n)) accesses to neighbours memory as easily. $\endgroup$ – jcm69 Feb 13 '19 at 21:28
  • $\begingroup$ @jcm69 can you explain or give the link for log_2(n) access time for VP memory access $\endgroup$ – Arayan Singh May 4 '19 at 16:41
  • $\begingroup$ That's just a general consideration about handling objects in computer science. When you are sure that objects follow power of 2 rules, they can be easily and safely managed by binary search trees, binary shifts etc. When you're not sure, well, you may have to make some additional tests and more complicated algo. Anyway, that's a little bit far from the initial question ;) $\endgroup$ – jcm69 May 4 '19 at 20:42
  • $\begingroup$ Does anyone know of any benchmarking experiments they could link to that show the % performance improvement in using powers of two against non-powers of two? $\endgroup$ – Keith Johnson Apr 15 at 13:45

The overall idea is to fit your mini-batch entirely in the the CPU/GPU. Since, all the CPU/GPU comes with a storage capacity in power of two, it is advised to keep mini-batch size a power of two.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.