# What is the advantage of keeping batch size a power of 2?

While training models in machine learning, why is it sometimes advantageous to keep the batch size to a power of 2? I thought it would be best to use a size that is the largest fit in your GPU memory / RAM.

This answer claims that for some packages, a power of 2 is better as a batch size. Can someone provide a detailed explanation / link to a detailed explanation for this? Is this true for all optimisation algorithms (gradient descent, backpropagation, etc) or only some of them?

This is a problem of alignment of the virtual processors (VP) onto the physical processors (PP) of the GPU. Since the number of PP is often a power of 2, using a number of VP different from a power of 2 leads to poor performance.
You can see the mapping of the VP onto the PP as a pile of slices of size the number of PP.
Say you've got 16 PP.
You can map 16 VP on them : 1 VP is mapped onto 1 PP.
You can map 32 VP on them : 2 slices of 16 VP, 1 PP will be responsible for 2 VP.
Etc. During execution, each PP will execute the job of the 1st VP he is responsible for, then the job of the 2nd VP etc.
If you use 17 VP, each PP will execute the job of their 1st PP, then 1 PP will execute the job of the 17th AND the other ones will do nothing (precised below).
This is due to the SIMD paradigm (called vector in the 70s) used by GPUs. This is often called Data Parallelism : all the PP do the same thing at the same time but on different data. See here.
More precisely, in the example with 17 VP, once the job of the 1st slice done (by all the PPs doing the job of their 1st VP), all the PP will do the same job (2nd VP), but only one has some data to work on.
Nothing to do with learning. This is only programming stuff.

• would it be more accurate to say that batch sizes should then be a multiple of the number of PP? That is, in your example we could map 16x3=48 VP to 16 PP? Feb 12 '19 at 21:47
• Yes. well... If you do the mapping VP -> PP, yourself, sure you're 100% right. If you use a library, and ask for 80 VP. I'm not sure. I'm not saying you're wrong. If the ratio is a power of 2, you can use very classical and easy optimizations.Think of memory accesses. If the # of upstairs neighbours of a VP is not a power of 2, say 5, the lib won't be able to use classical O(log_2(n)) accesses to neighbours memory as easily. Feb 13 '19 at 21:28
• @jcm69 can you explain or give the link for log_2(n) access time for VP memory access May 4 '19 at 16:41
• That's just a general consideration about handling objects in computer science. When you are sure that objects follow power of 2 rules, they can be easily and safely managed by binary search trees, binary shifts etc. When you're not sure, well, you may have to make some additional tests and more complicated algo. Anyway, that's a little bit far from the initial question ;) May 4 '19 at 20:42
• Does anyone know of any benchmarking experiments they could link to that show the % performance improvement in using powers of two against non-powers of two? Apr 15 '20 at 13:45

The overall idea is to fit your mini-batch entirely in the the CPU/GPU. Since, all the CPU/GPU comes with a storage capacity in power of two, it is advised to keep mini-batch size a power of two.

I just ran a quick experiment training yolov4-csp on coco with batch sizes 8 and 9 and found that per-image, batch sized 9 was slightly more efficient than 8. So at least with pytorch and relatively small batches on a modern GPU (2080Ti) it would seem that there is no negative performance impact of not using powers of 2 for batch sizes.

Measurements:

batch=9
4009/13143 batches
real    20m51.557s
per batch time = (20*60 + 51.557)/4009 = 0.312186829633325 seconds
per image time = 0.312186829633325 / 9 = 0.0347

batch=8
5037/14786 batches
real    23m51.666s
per batch time = (23*60 + 51.666)/5037 = 0.28422989874925547 seconds
per image time = 0.28422989874925547 / 8 = 0.0355 seconds

• Is it possible for you to report similar numbers for a higher batch size (63/64)? May 25 at 7:34
• @GaneshTata unfortunately I don't have the VRAM with this particular model, but maybe I can try with a different one. May 27 at 8:33

Have a look at this experimental data for average prediction speed per sample vs batch size. It very much underlines the points of the accepted answer of jcm69. It looks like this particular model (and its inputs) works optimal with batch sizes with multiples of 32 - note the line of sparse dots that is below the main line of dots. That might be different for other model-GPU combinations, but a power of two would be a safe bet for any combination.

The benchmark of ezekiel unfortunately isn't very telling because a batch size of 9 potentially allocates twice as much memory. That a batch size of 9 is therefore faster than a batch size of 8 is to be expected.

• v. interesting - do you have a link to the source? May 25 at 9:19
• I created this figure. Unfortunately, I cannot share the data or the model used. May 26 at 12:08
• ah ok, well I'm sure you'll understand if I'm a little sceptical about a <citation needed> graph. May 27 at 8:26
• I am very sorry that I can't share the model or data and I can totally understand. Nevertheless, I hope that the plot still makes the point, that to evaluate the question we need a more detailed experimental design. I would be very happy to see the same experiment with some freely available model and data. This was just the easiest way for me to produce some plot quickly. May 28 at 16:48