Dynamically remove data from training dataset

Question

I was wondering today if it would be a good approach to remove data dynamically from the training dataset when learning a neural network. Assuming a classification task, the approach would be something like

1. Train the network for an epoch.
2. Remove the elements in the training dataset with low entropy and correctly classified, ie: the network is sure of the prediction and the prediction is right.
3. Go back to 1.
4. From time to time check that the results previously discarded are still certain and correctly classified.

With this approach, one could reduce the training time while maintaining the accuracy of the model.

Does someone know if this is something that can be done? I know the approach is similar to active learning, where you only label elements with low certainty, but this approach is the other way around: we have labels for all the elements but we decide to discard some of them to make the training faster.

Answer 1

What you are describing is commonly called curriculum learning, non-uniform sampling of mini-batches during training. Curriculum learning has been shown to increase learning speed and improve final performance on test data.

Answer 2

The problem with your approach is catastrophic forgetting. If you train only on the samples not properly classified in the previous epoch, then, after training the following epoch, your model will perform poorly on the "easy data" because it overfitted the difficult one.

Also, the behavior of the model at the beginning of an epoch is much different than at the end, so, in order to achieve your proposed approach properly, you would need to perform inference again on the whole training data used at that epoch. This would certainly be very costly.

What you propose vaguely resembles a technique called "boosting", but instead of using a single model, boosting trains different models sequentially. The first model is trained on all data. The second model is trained focusing on the data where the first model performed badly (the data are given "weights" and the weights increase or decrease from one model to the next depending on the correctness of the predictions of the previous model). And so on. However, with boosting the first model is never touched again after training on all data, so it retains its classification power. Also, boosting is normally used with "weak" classifiers (i.e. not deep neural networks), although not necessarily.