I've constructed a CNN in Python using Numpy, which is trained with mini batch gradient descent for MNIST digit classification. When training with a batch size of 1, the time needed for 5 epochs is about 1200 s, which is only about 40% slower than with batch sizes 16 to 256. For batch size 4 and 8 it takes about 940 s and 890 s respectively. When i train it over 5 epochs with batch sizes 16, 32 or 256 the time is about 850 s (about the same for all three sizes).
I expected the time to shorten with bigger batch sizes, as seen with batch size 1 vs batch size 16, but why is the time the same for batch sizes 16, 32 and 256? The larger the batch size, the fewer the iterations. I would expect the computational cost of training a larger batch to be minimized with my vectorized code. None of the batch sizes seem to use any disk memory. Why is the time difference so small and why does the time stagnate at about 850 s for 5 epochs?