Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
While training models in machine learning, why is it sometimes advantageous to keep the batch size to a power of 2? I thought it would be best to use a size that is the largest fit in your GPU memory / RAM.
This answer claims that for some packages, a power of 2 is better as a batch size. Can someone provide a detailed explanation / link to a detailed explanation for this? Is this true for all optimisation algorithms (gradient descent, backpropagation, etc) or only some of them?
This is a problem of alignment of the virtual processors (VP) onto the physical processors (PP) of the GPU. Since the number of PP is often a power of 2, using a number of VP different from a power of 2 leads to poor performance.
You can see the mapping of the VP onto the PP as a pile of slices of size the number of PP.
Say you've got 16 PP.
You can map 16 VP on them : 1 VP is mapped onto 1 PP.
You can map 32 VP on them : 2 slices of 16 VP, 1 PP will be responsible for 2 VP.
Etc.
During execution, each PP will execute the job of the 1st VP he is responsible for, then the job of the 2nd VP etc.
If you use 17 VP, each PP will execute the job of their 1st PP, then 1 PP will execute the job of the 17th AND the other ones will do nothing (precised below).
This is due to the SIMD paradigm (called vector in the 70s) used by GPUs. This is often called Data Parallelism : all the PP do the same thing at the same time but on different data. See here.
More precisely, in the example with 17 VP, once the job of the 1st slice done (by all the PPs doing the job of their 1st VP), all the PP will do the same job (2nd VP), but only one has some data to work on.
Nothing to do with learning. This is only programming stuff.
I just ran a quick experiment training yolov4-csp on coco with batch sizes 8 and 9 and found that per-image, batch sized 9 was slightly more efficient than 8. So at least with pytorch and relatively small batches on a modern GPU (2080Ti) it would seem that there is no negative performance impact of not using powers of 2 for batch sizes.
Measurements:
batch=9
4009/13143 batches
real 20m51.557s
per batch time = (20*60 + 51.557)/4009 = 0.312186829633325 seconds
per image time = 0.312186829633325 / 9 = 0.0347
batch=8
5037/14786 batches
real 23m51.666s
per batch time = (23*60 + 51.666)/5037 = 0.28422989874925547 seconds
per image time = 0.28422989874925547 / 8 = 0.0355 seconds
Have a look at this experimental data for average prediction speed per sample vs batch size. It very much underlines the points of the accepted answer of jcm69. It looks like this particular model (and its inputs) works optimal with batch sizes with multiples of 32 - note the line of sparse dots that is below the main line of dots. That might be different for other model-GPU combinations, but a power of two would be a safe bet for any combination.
The benchmark of ezekiel unfortunately isn't very telling because a batch size of 9 potentially allocates twice as much memory. That a batch size of 9 is therefore faster than a batch size of 8 is to be expected.
The overall idea is to fit your mini-batch entirely in the the CPU/GPU. Since, all the CPU/GPU comes with a storage capacity in power of two, it is advised to keep mini-batch size a power of two.
