I was wondering today if it would be a good approach to remove data dynamically from the training dataset when learning a neural network. Assuming a classification task, the approach would be something like
- Train the network for an epoch.
- Remove the elements in the training dataset with low entropy and correctly classified, ie: the network is sure of the prediction and the prediction is right.
- Go back to 1.
- From time to time check that the results previously discarded are still certain and correctly classified.
With this approach, one could reduce the training time while maintaining the accuracy of the model.
Does someone know if this is something that can be done? I know the approach is similar to active learning, where you only label elements with low certainty, but this approach is the other way around: we have labels for all the elements but we decide to discard some of them to make the training faster.