For the purpose of this answer, let’s assume your training data is more or less perfectly representative of the distribution of the real world data it was sampled from.
Let’s think about mini batch gradient descent. When you have a small batch size, each training step uses a small batch of training examples which is probably not able to accurately represent the wider data set. This means the calculated error gradient is not quite accurate, and effectively the parameters of the model are updated as though the small set of training examples is totally representative of the underlying distribution. If the mini batch is especially far from the underlying distribution, the mini batches that come after it will on average bring the model back towards the correct distribution.
We use a small learning rate to ensure that only small updates are made at each training step, which essentially reduces the risk of the model parameters going wildly off course in response to a weird mini batch and therefore keeps the model’s descent nearer to an imaginary “perfect” descent down the error gradient (because on average a mini batch will represent the exact distribution of your training data).
If we use bigger batches, they are more likely to be representative of the underlying distribution, so we can safely use a higher learning rate. As our model starts to converge, it should still keep moving towards a true error minimum and not be thrown off by weird batches. We might use learning rate scheduling to reduce the learning rate as training goes on.
Stochastic gradient descent is equivalent to mini batch gradient descent with batch size 1, so we’re in danger of being thrown off our ideal descent by weird observations if our learning rate is too high. This might throw us off the path to a local minimum and onto a path towards the global minimum (or at least a better local minimum), but it might also do the opposite. Therefore, this is not a good justification of using a very small batch size.
Of course, sometimes we’re forced to use small batch sizes because our data is too big to fit many observations in memory at once. Otherwise, a bigger batch size is better.