From visual inspection of a sub-portion of my data, I estimate that around 5-6% of the labels are incorrect.

My classifier still performs well and when I take prediction probabilities that are above .95 for a given class that are in contrast to the actual label, I find that 92% of the time the classifiers prediction is correct. So, for example, a review text with the label positive that has an estimated probability of >= .95 of being negative, is in fact negative most of the time.

Can I therefore use the most confident predictions to correct some of the noisy labels? And not retrain on the corrected data, but use the probabilities to correct the validation and test sets to get more accurate final performance and also to calibrate the estimates (as noisy labels may be particularly harmful to calibration).

EDIT: Follow up to answer below.

When I train on the uncorrected labels and then predict the test set I get:

When I train on the corrected labels for the corrected test set I get:

Likewise, the model trained on the uncorrected data performs better on the original uncorrected test set than the model trained with the corrected data. Somehow removing only incorrect labels with high confidence from the train set appears to decrease performance on unseen data. I would probably have to remove all incorrect labels and not only those identified through high confidence.

From visual inspection of a sub-portion of my data, I estimate that around 5-6% of the labels are incorrect.

My classifier still performs well and when I take prediction probabilities that are above .95 for a given class that are in contrast to the actual label, I find that 92% of the time the classifiers prediction is correct. So, for example, a review text with the label positive that has an estimated probability of >= .95 of being negative, is in fact negative most of the time.

Can I therefore use the most confident predictions to correct some of the noisy labels? And not retrain on the corrected data, but use the probabilities to correct the validation and test sets to get more accurate final performance and also to calibrate the estimates (as noisy labels may be particularly harmful to calibration).

EDIT: Follow up to answer below.

When I train on the uncorrected labels and then predict the test set I get:

When I train on the corrected labels for the corrected test set I get:

Likewise, the model trained on the uncorrected data performs better on the original uncorrected test set than the model trained with the corrected data. Somehow removing only incorrect labels with high confidence from the train set appears to decrease performance on unseen data. I would probably have to remove all incorrect labels and not only those identified through high confidence.

EDIT EDIT: After experimenting for a while I've reached the following conclusion:

It seems that predictions can be used for label corrections, but a different estimator should be used to try and determine which labels are incorrect. When I used the same estimator, despite most of the label corrections being valid (I determined 92-93% based on visual inspection of a 500 sample subset), it nevertheless caused the new prediction estimates to be biased. The new estimates were overly confident (drastically tended towards zero and one). This is either due to the correction, or it may be due to too little noise in the dataset (I considered the possibility that the noise is actually helping the estimator not to overfit. Neural networks have been found to be poorly calibrated, the author of this article suggests that overestimation may in fact be a form of overfitting).

From visual inspection of a sub-portion of my data, I estimate that around 5-6% of the labels are incorrect.

My classifier still performs well and when I take prediction probabilities that are above .95 for a given class that are in contrast to the actual label, I find that 92% of the time the classifiers prediction is correct. So, for example, a review text with the label positive that has an estimated probability of >= .95 of being negative, is in fact negative most of the time.

Can I therefore use the most confident predictions to correct some of the noisy labels? And not retrain on the corrected data, but use the probabilities to correct the validation and test sets to get more accurate final performance and also to calibrate the estimates (as noisy labels may be particularly harmful to calibration).

From visual inspection of a sub-portion of my data, I estimate that around 5-6% of the labels are incorrect.

My classifier still performs well and when I take prediction probabilities that are above .95 for a given class that are in contrast to the actual label, I find that 92% of the time the classifiers prediction is correct. So, for example, a review text with the label positive that has an estimated probability of >= .95 of being negative, is in fact negative most of the time.

Can I therefore use the most confident predictions to correct some of the noisy labels? And not retrain on the corrected data, but use the probabilities to correct the validation and test sets to get more accurate final performance and also to calibrate the estimates (as noisy labels may be particularly harmful to calibration).

EDIT: Follow up to answer below.

When I train on the uncorrected labels and then predict the test set I get:

When I train on the corrected labels for the corrected test set I get:

Likewise, the model trained on the uncorrected data performs better on the original uncorrected test set than the model trained with the corrected data. Somehow removing only incorrect labels with high confidence from the train set appears to decrease performance on unseen data. I would probably have to remove all incorrect labels and not only those identified through high confidence.