This question is specifically regarding imperfect labels.
I'd like to understand the theoretical and practical implications of removing examples from the training set based on information obtained outside the performance window.
For example, let's say we're looking to identify fraudulent accounts in our data, so we define our label as "is this account flagged as fraud in the next 30 days"? Despite using this, for many accounts we have performance data beyond 30 days, and for some substantial subset of these accounts it turns out they're flagged after more than 30 days (and let's just assume for this exercise that they were already fraudulent when we sampled for the training dataset). Keeping these examples in the training data introduces noise, which can significantly impact the model's performance if very precise labeling is necessary.
What is wrong with removing these examples from our training data (since we don't want to change the label, but know that the label is incorrect)? Given that we have a completely separate test set (out of time and out of sample).