Say I am training a neural network and can fit all my data into memory. Are there any benefits to using mini batches with SGD in this case? Or is batch training with the full gradient always superior when possible?

Also, it seems like many of the more modern optimization algorithms (RMSProp, Adam, etc.) were designed with SGD in mind. Are these methods still superior to standard gradient descent (with momentum) with the full gradient available?


On large datasets, SGD can converge faster than batch training because it performs updates more frequently. We can get away with this because the data often contains redundant information, so the gradient can be reasonably approximated without using the full dataset. Minibatch training can be faster than training on single data points because it can take advantage of vectorized operations to process the entire minibatch at once. The stochastic nature of online/minibatch training can also make it possible to hop out of local minima that might otherwise trap batch training.

One reason to use batch training is cases where the gradient can't be approximated using individual points/minibatches (e.g. where the loss function can't be decomposed as a sum of errors for each data point). This isn't an issue for standard classification/regression problems.

I don't recall seeing RMSprop/Adam/etc. compared to batch gradient descent. But, given their potential advantages over vanilla SGD, and the potential advantages of vanilla SGD over batch gradient descent, I imagine they'd compare favorably.

Of course, we have to keep the no free lunch theorem in mind; there must exist objective functions for which each of these optimization algorithms performs better than the others. But, there's no guarantee whether or not these functions pertain to the set of practically useful, real-world learning problems.

  • 1
    $\begingroup$ Thanks, intuitively this makes a lot of sense. I was also thinking maybe SGD could help prevent overfitting. A little bit of randomness in model training usually seems to help the cause. $\endgroup$ – Nitro Jan 30 '17 at 16:03
  • $\begingroup$ My understanding is: Compared to batch gradient descent, SGD can indeed improve generalization performance for large-scale problems (i.e. those constrained by computation time rather than data availability). The reason for this is that SGD can process more examples within the available computation time. This reasoning doesn't apply to small scale problems that are limited by data availability but not computation time. See: Bottou and Le Cun (2004). Large scale online learning. Bottou and Bousquet (2008). The tradeoffs of large scale learning. $\endgroup$ – user20160 Jan 31 '17 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.