63
$\begingroup$

I am about to train a big LSTM network with 2-3 million articles and am struggling with Memory Errors (I use AWS EC2 g2x2large).

I found out that one solution is to reduce the batch_size. However, I am not sure if this parameter is only related to memory efficiency issues or if it will effect my results. As a matter of fact, I also noticed that batch_size used in examples is usually as a power of two, which I don't understand either.

I don't mind if my network takes longer to train, but I would like to know if reducing the batch_size will decrease the quality of my predictions.

Thanks.

$\endgroup$
2
  • 2
    $\begingroup$ This question is not specific to keras. I think the general consesus is that smaller sample sizes converge slower but are less prone to getting stuck in the local minima $\endgroup$
    – Alex
    Oct 23, 2017 at 10:06
  • 1
    $\begingroup$ I have seen cases where too large a batch size can prevent convergence even with same number of training epochs. $\endgroup$ Oct 14, 2018 at 13:04

4 Answers 4

75
$\begingroup$

After one and a half years, I come back to my answer because my previous answer was wrong.

Batch size impacts learning significantly. What happens when you put a batch through your network is that you average the gradients. The concept is that if your batch size is big enough, this will provide a stable enough estimate of what the gradient of the full dataset would be. By taking samples from your dataset, you estimate the gradient while reducing computational cost significantly. The lower you go, the less accurate your esttimate will be, however in some cases these noisy gradients can actually help escape local minima. When it is too low, your network weights can just jump around if your data is noisy and it might be unable to learn or it converges very slowly, thus negatively impacting total computation time.

Another advantage of batching is for GPU computation, GPUs are very good at parallelizing the calculations that happen in neural networks if part of the computation is the same (for example, repeated matrix multiplication over the same weight matrix of your network). This means that a batch size of 16 will take less than twice the amount of a batch size of 8.

In the case that you do need bigger batch sizes but it will not fit on your GPU, you can feed a small batch, save the gradient estimates and feed one or more batches, and then do a weight update. This way you get a more stable gradient because you increased your virtual batch size.

$\endgroup$
5
  • $\begingroup$ I can't afford 32 but I can afford 16. However, I noticed that it is too slow. Do you think I should try some values between 16-32 or stick with 16? $\endgroup$ Jul 2, 2016 at 8:21
  • $\begingroup$ I would try and time some values. Every epoch should be around the same time so that won't take too long. Try 17 first to see if it's faster or slower because I'm interested in this, given that this power of 2 depends on GPU and/or backend of Keras. But I think just filling it to the brim is likely best $\endgroup$ Jul 2, 2016 at 8:23
  • $\begingroup$ Does batch_size affects the memory size? It shouldn't be so, as the samples could be processed one at the time, and then the gradient updates should be averaged, so it shouldn't increase memory size. $\endgroup$
    – Nathan B
    Jan 15, 2018 at 12:11
  • $\begingroup$ batch_size definetely affects memory size, because it will normally attempt to process them parallel (this makes GPUs very efficient). If you want to process them sequentially and average the gradients this is possible but each of the individual samples would then be considered a batch, and you delay the gradient updates $\endgroup$ Jan 15, 2018 at 12:27
  • $\begingroup$ Is there also such aspect? When your batch size is smaller, changes flow faster through network. E.g. after some neiron on the 2nd layer starts to be more or less adequate, recognition of some low-level features on the 1nd layer improves and then other neirons on the 2nd layer start to catch some useful signal from them... I have image recognition convolution network in mind, but maybe this can fit to other architectures $\endgroup$
    – Mikhail M
    Apr 2, 2020 at 16:10
19
$\begingroup$

I feel the accepted answer is possibly wrong. There are variants in Gradient Descent Algorithms.

  1. Vanilla Gradient Descent: Here the Gradient is being calculated on all the data points at a single shot and the average is taken. Hence we have a smoother version of the gradient takes longer time to learn.

  2. Stochastic Gradient Descent : Here one-data point at a time hence the gradient is aggressive (noisy gradients) hence there is going to be lot of oscillations ( we use Momentum parameters - e.g Nesterov to control this). So there is a chance that your oscillations can make the algorithm not reach a local minimum.(diverge).

  3. Mini-Batch Gradient Descent: Which takes the perks of both the previous ones averages gradients of a small batch. Hence not too aggressive like SGD and allows Online Learning which Vanilla GD never allowed.

The smaller the Mini-Batch the better would be the performance of your model (not always) and of course it has got to do with your epochs too faster learning. If you are training on large dataset you want faster convergence with good performance hence we pick Batch-GD's.

SGD had fixed learning parameter hence we start other Adaptive Optimizers like Adam, AdaDelta, RMS Prop etc which changes the learning parameter based on the history of Gradients.

$\endgroup$
1
  • 1
    $\begingroup$ I agree there is no rule regarding the batch-size parameter. But this statement - "The smaller the Mini-Batch the better would be the performance of your model" - is contrary to the general rule. You generally want to maximize the batch-size $\endgroup$ Nov 21, 2018 at 22:35
8
$\begingroup$

Oddly enough, I found that larger batch sizes with keras require more epochs to converge.

For example, the output of this script based on keras' integration test is

epochs 15   , batch size 16   , layer type Dense: final loss 0.56, seconds 1.46
epochs 15   , batch size 160  , layer type Dense: final loss 1.27, seconds 0.30
epochs 150  , batch size 160  , layer type Dense: final loss 0.55, seconds 1.74

Related

Using too large a batch size can have a negative effect on the accuracy of your network during training since it reduces the stochasticity of the gradient descent.

Edit: most of the times, increasing batch_size is desired to speed up computation, but there are other simpler ways to do this, like using data types of a smaller footprint via the dtype argument, whether in keras or tensorflow, e.g. float32 instead of float64

$\endgroup$
1
  • 2
    $\begingroup$ With bigger batches (and therefore fewer per epoch) you will have fewer gradient updates per epoch. "epoch" is just ML jargon for "one pass through your data during training". If you are trying to speed up training, measure wall time and ignore epochs. $\endgroup$ May 23, 2019 at 10:34
3
$\begingroup$

A couple of papers have been published showing – and conventional wisdom in 2020 seems to still be persuaded— that, as Yann LeCun put, large batches are bad for your health.

Two relevant papers are

and

which offers possible reasons. To paraphrase badly, big batches are likely to get stuck in local (“sharp”) minima, small batches not. There is some interplay with choice of learning rate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.