Can I use confident predictions to correct incorrect labels?

Question

From visual inspection of a sub-portion of my data, I estimate that around 5-6% of the labels are incorrect.

My classifier still performs well and when I take prediction probabilities that are above .95 for a given class that are in contrast to the actual label, I find that 92% of the time the classifiers prediction is correct. So, for example, a review text with the label positive that has an estimated probability of >= .95 of being negative, is in fact negative most of the time.

Can I therefore use the most confident predictions to correct some of the noisy labels? And not retrain on the corrected data, but use the probabilities to correct the validation and test sets to get more accurate final performance and also to calibrate the estimates (as noisy labels may be particularly harmful to calibration).

EDIT: Follow up to answer below.

When I train on the uncorrected labels and then predict the test set I get:

When I train on the corrected labels for the corrected test set I get:

Likewise, the model trained on the uncorrected data performs better on the original uncorrected test set than the model trained with the corrected data. Somehow removing only incorrect labels with high confidence from the train set appears to decrease performance on unseen data. I would probably have to remove all incorrect labels and not only those identified through high confidence.

EDIT EDIT: After experimenting for a while I've reached the following conclusion:

It seems that predictions can be used for label corrections, but a different estimator should be used to try and determine which labels are incorrect. When I used the same estimator, despite most of the label corrections being valid (I determined 92-93% based on visual inspection of a 500 sample subset), it nevertheless caused the new prediction estimates to be biased. The new estimates were overly confident (drastically tended towards zero and one). This is either due to the correction, or it may be due to too little noise in the dataset (I considered the possibility that the noise is actually helping the estimator not to overfit. Neural networks have been found to be poorly calibrated, the author of this article suggests that overestimation may in fact be a form of overfitting).

Answer 1

At the moment, you don't really know the performance of your model because you have quite a few wrong labels in your test set.

You mentioned that you want to use the new probabilities to correct the validation and test sets. However, if you do that, you will of course get higher results because you are using the labels coming from your own model. However, if you verify by hand than it is ok.

Therefore, what I would suggest is:

1. Train the classifier with all your data. Don't do any splits.
2. After training, pass all your data through the model. Separate all the data that the model gets wrong.
3. Manually check all the mistakes. You will have a few scenarios: The model gets is wrong but the label was correct) in this case, leave the label as it was. The model gets it wrong but was wrong) this are the interesting cases for you, where the model is wrong but in fact it is correct since the original labels were wrong. Correct these.
4. Now perform train/validation/test splits.
5. Retrain your model with the train/validation splits.
6. Test your model with the test split.

Answer 2

Both your question Danyal and Bruno's answer have been addressed by the field of confident learning, a field I worked on with Isaac Chuang during my PhD at MIT.

The Confident Learning paper (Northcutt, Jiang, Lu (2021) Journal of AI Research) should answer all of your questions: https://arxiv.org/abs/1911.00068

(very brief summary of Confident Learning that answers your questions

• Confident Learning is a field of ML for finding label errors, estimating uncertainty in dataset labels, learning with noisy labels, and dataset curation.
• Confident learning is proven to exactly find label errors for realistic settings where the model produces imperfect probabilities for every datapoint and every class (but only within a range... as your model's performance deteriorates on perfectly-labeled data, so will your ability to use that model to find label errors).
• Your approach to look at confident predictions works, but use these to find the label errors and train on clean data instead of re-labeling. When you re-label using the predictions without a human-in-the-loop, you usually introduce bias by the model (as you noticed).
• You may also have realized that you don't know (for an arbitrary dataset) what is the threshold of confidence to determine that something is confident labeled. There is a proven solution for the right threshold (use cleanlab below).

Why you need to clean label errors out of the train and test set:

• Did you know there are millions of label errors in the ten most commonly used ML test sets (yes... test sets... the ones we've been benchmarking on).
• View them here: https://labelerrors.com
• NeurIPS 2021 paper (nominated for best paper award): https://arxiv.org/abs/2103.14749
• Takeaway: use tools like cleanlab to produce a clean test set otherwise you can't trust your benchmarks.

Everything shared above is implemented in the cleanlab open-source package for you. I starting writing this library in grad school then I raised funds to build it out into the standard package to improve any dataset for ML.

Find label issues in your dataset in 1 line of code

from cleanlab.classification import CleanLearning
from cleanlab.filter import find_label_issues

# Option 1 - works with sklearn-compatible models - just input the data and labels ツ
label_issues_info = CleanLearning(clf=sklearn_compatible_model).find_label_issues(data, labels)

# Option 2 - works with ANY ML model - just input the model's predicted probabilities
ordered_label_issues = find_label_issues(
    labels=labels,
    pred_probs=pred_probs,  # out-of-sample predicted probabilities from any model
    return_indices_ranked_by='self_confidence',
)

Train a model as if the dataset did not have errors -- 3 lines of code

from sklearn.linear_model import LogisticRegression
from cleanlab.classification import CleanLearning

cl = CleanLearning(clf=LogisticRegression())  # any sklearn-compatible classifier
cl.fit(train_data, labels)

# Estimate the predictions you would have gotten if you trained without mislabeled data.
predictions = cl.predict(test_data)

• cleanlab: https://github.com/cleanlab/cleanlab
• Documentation and runnable tutorials for cleanlab: https://docs.cleanlab.ai/