I find it best to ask my question in terms of cross-validation. Here it goes:
Suppose a binary classification problem, for which cross-validation has been applied for a certain learning algorithm. Let's say that both the CV train error and CV test error are at 90% accuracy, indicating a good fit. Since this performance is acceptable for our problem, we combine the training and validation set into a final full dataset, and train the final model. For the final dataset, only training error is available, which suppose for our example will be 92%.
Now, for the question: Knowing that the final model has achieved a 92% accuracy, does it serve any purpose to keep the 8% of missclasified examples in the final dataset? Since these examples can't be learned, why not remove them and retrain the final model with only the 92% of the data that can be learned?
- To the best of my knowledge, the aforementioned removal of 8% in the above example is not a standard practice in modelling. Yet, I wonder what is the value of keeping examples that are not learned.
- For completion, assume there is also another independent test set, to evaluate the final model.