Powers of 2 for batch_size in model fit in deep learning

Question

I am currently reading Deep Learning with Python by Francois Chollet, the author of Keras, and in one of his definitions for Mini-batch, he explains that the power of 2 for the batch_size is due to memory allocations in gpu/ Could anyone elaborate on this?

Mini-batch or batch—A small set of samples (typically between 8 and 128) that are processed simultaneously by the model. The number of samples is often a power of 2, to facilitate memory allocation on GPU. When training, a mini-batch is used to compute a single gradient-descent update applied to the weights of the model.

Answer 1

It's not just on GPU, you want to have a small power of two as submultiple for CPU as well.

The reason is to allocate the processors on the GPU (or the SIMD registers) as best as possibles. This is directly linked to the warp sizes and what the GPU vendors offer for maximum block sizes,

See also this thread on SO which what the warp size is and its relation to best occupancy.