1
$\begingroup$

I was wondering today if it would be a good approach to remove data dynamically from the training dataset when learning a neural network. Assuming a classification task, the approach would be something like

  1. Train the network for an epoch.
  2. Remove the elements in the training dataset with low entropy and correctly classified, ie: the network is sure of the prediction and the prediction is right.
  3. Go back to 1.
  4. From time to time check that the results previously discarded are still certain and correctly classified.

With this approach, one could reduce the training time while maintaining the accuracy of the model.

Does someone know if this is something that can be done? I know the approach is similar to active learning, where you only label elements with low certainty, but this approach is the other way around: we have labels for all the elements but we decide to discard some of them to make the training faster.

$\endgroup$

2 Answers 2

0
$\begingroup$

What you are describing is commonly called curriculum learning, non-uniform sampling of mini-batches during training. Curriculum learning has been shown to increase learning speed and improve final performance on test data.

$\endgroup$
2
  • 1
    $\begingroup$ But normally on curriculum learning, you decide a priori which data to feed the model training based on some criteria that do not involve the model itself, right? And usually, curriculum learning is incremental, not the other way around, like the approach proposed in the question. $\endgroup$
    – noe
    Feb 8 at 7:56
  • $\begingroup$ Yes, the point is to dynamically tune the training dataset depending on how the training process is going. The idea is to use "online" information to decide which training examples to use next. $\endgroup$
    – alexmolas
    Feb 8 at 8:13
0
$\begingroup$

The problem with your approach is catastrophic forgetting. If you train only on the samples not properly classified in the previous epoch, then, after training the following epoch, your model will perform poorly on the "easy data" because it overfitted the difficult one.

Also, the behavior of the model at the beginning of an epoch is much different than at the end, so, in order to achieve your proposed approach properly, you would need to perform inference again on the whole training data used at that epoch. This would certainly be very costly.

What you propose vaguely resembles a technique called "boosting", but instead of using a single model, boosting trains different models sequentially. The first model is trained on all data. The second model is trained focusing on the data where the first model performed badly (the data are given "weights" and the weights increase or decrease from one model to the next depending on the correctness of the predictions of the previous model). And so on. However, with boosting the first model is never touched again after training on all data, so it retains its classification power. Also, boosting is normally used with "weak" classifiers (i.e. not deep neural networks), although not necessarily.

$\endgroup$
2
  • 1
    $\begingroup$ does boosting really work like that? I thought all the weak classifiers were trained using all data, and each classifier tried to correct the error of the previous classifiers. $\endgroup$
    – alexmolas
    Feb 8 at 8:11
  • $\begingroup$ Yes, they are all trained on all data, but the data misclassified by a classifier is given more "importance" (increased weight) than the correctly classified data (decreased weight) for the training of the next classifier. I will reformulate that paragraph in the answer. Thank you! $\endgroup$
    – noe
    Feb 8 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.