I have a text-classification problem with a lot of training data. Running cross-validation takes a lot of time - several days or even weeks. In order to make the system more responsive, I am thinking of the following scheme:

  • Train a network on instances 1,...,100.
  • Test it on instances 101,...200; output the accuracy.
  • Train the existing network on instances 101,...200;
  • Test it on instances 201,...300; output the accuracy.
  • And so on.

Ideally, I would like that:

  • Training on each additional 100 instances will take a constant time (i.e, I will not have to re-train the network on all previous instances);
  • The trained network will have the combined "wisdom" of all previous instances (so its accuracy will tend to improve with time).

Is this possible to do with standard deep-learning tools (e.g. dynet)?


You can achieve that by training the network using batches of 100 examples. In this case, you train on a batch and update the neural network parameters, you predict on the second batch to estimate the accuracy and then you use the examples of this second batch to continue training the network and so on and so forth.

Batch training (a form of online learning) is implemented on every framework, see for instances here for Keras.

  • $\begingroup$ So I train on the first batch for e.g. 100 iterations, then train on the second batch for 100 iterations, and so on? Doesn't the later trainings "delete" the effect of the earlier trainings? $\endgroup$ – Erel Segal-Halevi Jan 3 '18 at 14:42
  • $\begingroup$ True. It will actually overfit them and the generalization performance will drop considerably. I would suggest training for one iteration, moving to the next batch and repeat. Once the whole dataset is consumed, you start from the beginning. $\endgroup$ – geompalik Jan 3 '18 at 15:14
  • $\begingroup$ So I have to keep all training instances in memory, and train from the beginning each time? $\endgroup$ – Erel Segal-Halevi Jan 3 '18 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.