Online vs minibatch training for speed

Question

If I do online learning in a setting where I have a HUGE amount of data, is that faster than doing minibatch learning (even if I optimize my batch size for GPU use, that is, use a multiple of 32 examples per minibatch)?

Details:

I have 12600 time series examples, each with 24 time steps, and each time step has 972196 binary labels. This is a multilabel problem.

Assuming float32 numbers:

• loading the entire dataset should take about 1095 GB (32*12600*24*972196) = 9.4x10^12 bits)
• loading a minibatch of size 32 should take about 2.7 GB (32*32*24*972196 bits)
• loading one training example should take about 89 MB (32*24*972196 bits)

I'm currently using an LSTM with online learning, and it's taking about 10 seconds per training example. I'm looking to speed this up.

Other related answers:

One answer to this question implies that minibatch learning will not be helpful when there are millions of features or millions of observations (I think they're just talking about the memory limitations).

Another question is related to what I'm asking, but on the implementation side of things.

I've read this question and the answers, and it's very good but doesn't quite answer my question.

Answer 1

The training one-example-at-time (what you can online learning) will yield worse performance on the evaluation metrics than training with many-examples-at-time. One-example-at-time will encourage the model to overfit to each and every data example. Increasing batch size will encourage the model to learn generalizable patterns.

You'll have to balance training speed with evaluation metric performance.

As far speeding-up your training, there are many possibilities:

• Reduce numerical precision from float32
• Reduce the number of features
• Get a bigger or better computer
• Distribute the workload across a cluster.