Which samples to inspect in a noisy labeled classification task?

Question

I have a dataset with noisy labels on which I train a binary classifier. Inspecting the loss I see some samples were misclassified with high confidence and others were classified with indecision around 0.5 confidence.

If I have a budget to inspect wrongly predicted samples and potentially relabel them, is it better to pick the ones where the model was certain but got it wrong or the ones where the model is uncertain?

---

On the one hand, if I select the samples where the model has high confidence it could either be a sample which was classified correctly but has a bad label or the sample was indeed misclassified. The latter is a very valuable sample, because the classifier misclassified with high confidence. The former however is useless, as the model was in fact correct.

On the other hand, if I select the samples where the model was uncertain I will always gain insights (samples might have 50% bad label ratio, cannot learn label from sample, etc.) but it might not be as valuable as the high confidence false-positives/false-negatives.

Which group should I pick?

Answer 1

The most common goal of machine learning is to learn the best overall model for the entire dataset. Given that goal, your time would be best spent on samples where the model is very confident but wrong. Correcting these errors will have the greatest effect on decreasing training error and the greatest chance of increasing generalization.

Other paths might be useful for exploratory data analysis but will not as directly improve model performance. For example if the model is uncertain about a datapoint, then set a threshold to not make a prediction.