My training data is very huge and it's impossible to load all of it at once even into main memory. So I'm loading a few blocks (subset) of data and training till convergence, then proceeding to next subset and training till convergence and so on. Is it the right approach ?

The model performance kind of remains the same even when training on a new subset of data.

Is this method fundamentally wrong, why ?


Training until convergence on a subset of data and starting again on another subset is not a good idea.

Gradients of loss will have high variance over your batches and so optimizing over it will not be useful (you basically start at a far new point on your loss for the next subset).

Instead, the good way to go in batch-training methods is to iteratively make training steps on differents subsets of your data until a global convergence. Your may hear about an epoch, which consists of training steps on subsets until the overall data has been seen. You see how convergence is over the epochs and not over one subset.

| improve this answer | |
  • $\begingroup$ Yes makes sense, then how does transfer learning works ? Or say you already have a trained model - you get new data in future - is it not correct to re-train the model on new data ? Should we do it from scratch ? $\endgroup$ – wolframalpha Jan 4 '19 at 10:48
  • 1
    $\begingroup$ When you do transfer learning, you often don't train the entire model again by fixing some of the parameters you already have. Also, you form the hypothesis that the new data follows a close distribution to the old data and that your model is enough robust thanks to good amounts of old training data to only fine tune on small new data. Training to convergence between batches would then be similar to do transfer learning at each new subset but unless the subset is big (which does not make sense) the training will be of poor quality. $\endgroup$ – Elliot Jan 4 '19 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.