regarding the inclusion of missclassified data in final training

Question

I find it best to ask my question in terms of cross-validation. Here it goes:

Suppose a binary classification problem, for which cross-validation has been applied for a certain learning algorithm. Let's say that both the CV train error and CV test error are at 90% accuracy, indicating a good fit. Since this performance is acceptable for our problem, we combine the training and validation set into a final full dataset, and train the final model. For the final dataset, only training error is available, which suppose for our example will be 92%.

Now, for the question: Knowing that the final model has achieved a 92% accuracy, does it serve any purpose to keep the 8% of missclasified examples in the final dataset? Since these examples can't be learned, why not remove them and retrain the final model with only the 92% of the data that can be learned?

Notes

• To the best of my knowledge, the aforementioned removal of 8% in the above example is not a standard practice in modelling. Yet, I wonder what is the value of keeping examples that are not learned.
• For completion, assume there is also another independent test set, to evaluate the final model.

Answer 1

Since these examples can't be learned, why not remove them and retrain the final model with only the 92% of the data that can be learned?

In general, I think this is a bad idea for the reasons below. This being said, the only sure way to know with a particular dataset is to experiment.

• This would modify the distribution of the data. If the errors tend to happen more often with a particular class or a specific combination of features (this is quite likely), these cases would not be seen anymore by the model. Sometimes modifying the distribution can lead to better performance (e.g. when using resampling), but it can also do the opposite. So this is a bias with unknown effects on the performance of the model.
• A more serious problem is that these error cases are likely to be important for the model. Typically errors happen with borderline cases, and these cases are usually very important for the model to learn the optimal way to separate the classes. By removing them it might be easier for the model to separate the classes during training, but that's not a good thing since it won't have all the information it needs in order to minimize errors. As a consequence it's likely that it won't find the right optimum and therefore will make more errors.