If I do online learning in a setting where I have a HUGE amount of data, is that faster than doing minibatch learning (even if I optimize my batch size for GPU use, that is, use a multiple of 32 examples per minibatch)?


I have 12600 time series examples, each with 24 time steps, and each time step has 972196 binary labels. This is a multilabel problem.

Assuming float32 numbers:

  • loading the entire dataset should take about 1095 GB (32*12600*24*972196) = 9.4x10^12 bits)
  • loading a minibatch of size 32 should take about 2.7 GB (32*32*24*972196 bits)
  • loading one training example should take about 89 MB (32*24*972196 bits)

I'm currently using an LSTM with online learning, and it's taking about 10 seconds per training example. I'm looking to speed this up.

Other related answers:

One answer to this question implies that minibatch learning will not be helpful when there are millions of features or millions of observations (I think they're just talking about the memory limitations).

Another question is related to what I'm asking, but on the implementation side of things.

I've read this question and the answers, and it's very good but doesn't quite answer my question.


1 Answer 1


The training one-example-at-time (what you can online learning) will yield worse performance on the evaluation metrics than training with many-examples-at-time. One-example-at-time will encourage the model to overfit to each and every data example. Increasing batch size will encourage the model to learn generalizable patterns.

You'll have to balance training speed with evaluation metric performance.

As far speeding-up your training, there are many possibilities:

  • Reduce numerical precision from float32
  • Reduce the number of features
  • Get a bigger or better computer
  • Distribute the workload across a cluster.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.