6
$\begingroup$

From visual inspection of a sub-portion of my data, I estimate that around 5-6% of the labels are incorrect.

My classifier still performs well and when I take prediction probabilities that are above .95 for a given class that are in contrast to the actual label, I find that 92% of the time the classifiers prediction is correct. So, for example, a review text with the label positive that has an estimated probability of >= .95 of being negative, is in fact negative most of the time.

Can I therefore use the most confident predictions to correct some of the noisy labels? And not retrain on the corrected data, but use the probabilities to correct the validation and test sets to get more accurate final performance and also to calibrate the estimates (as noisy labels may be particularly harmful to calibration).

EDIT: Follow up to answer below.

When I train on the uncorrected labels and then predict the test set I get:

Corrected Test Set - Model trained uncorrected

When I train on the corrected labels for the corrected test set I get:

Corrected Test Set - Model trained corrected

Likewise, the model trained on the uncorrected data performs better on the original uncorrected test set than the model trained with the corrected data. Somehow removing only incorrect labels with high confidence from the train set appears to decrease performance on unseen data. I would probably have to remove all incorrect labels and not only those identified through high confidence.

EDIT EDIT: After experimenting for a while I've reached the following conclusion:

It seems that predictions can be used for label corrections, but a different estimator should be used to try and determine which labels are incorrect. When I used the same estimator, despite most of the label corrections being valid (I determined 92-93% based on visual inspection of a 500 sample subset), it nevertheless caused the new prediction estimates to be biased. The new estimates were overly confident (drastically tended towards zero and one). This is either due to the correction, or it may be due to too little noise in the dataset (I considered the possibility that the noise is actually helping the estimator not to overfit. Neural networks have been found to be poorly calibrated, the author of this article suggests that overestimation may in fact be a form of overfitting).

$\endgroup$
  • $\begingroup$ My take: since you don't know the truth, you train your model on "wrong" data and the model will learn this "wrong" things. So frankly: I don't think that looking at the model outcome will help you to identify wrong labels. Erroneous is erroneous data. $\endgroup$ – Peter Jan 7 at 14:00
  • $\begingroup$ I have an eyeball validation set of 300 samples, and from this I've established that ~55% of the incorrect labels are actually correctly predicted by the classifier. Usually with high confidence. The remaining 45% are incorrectly labeled and the model predicted the incorrect label. Interestingly, these are generally much less confidently predicted, even though the prediction is consistent with the label. $\endgroup$ – Danyal Andriano Jan 7 at 18:57
1
$\begingroup$

At the moment, you don't really know the performance of your model because you have quite a few wrong labels in your test set.

You mentioned that you want to use the new probabilities to correct the validation and test sets. However, if you do that, you will of course get higher results because you are using the labels coming from your own model. However, if you verify by hand than it is ok.

Therefore, what I would suggest is:

  1. Train the classifier with all your data. Don't do any splits.
  2. After training, pass all your data through the model. Separate all the data that the model gets wrong.
  3. Manually check all the mistakes. You will have a few scenarios: The model gets is wrong but the label was correct) in this case, leave the label as it was. The model gets it wrong but was wrong) this are the interesting cases for you, where the model is wrong but in fact it is correct since the original labels were wrong. Correct these.
  4. Now perform train/validation/test splits.
  5. Retrain your model with the train/validation splits.
  6. Test your model with the test split.
| improve this answer | |
$\endgroup$
  • $\begingroup$ thanks for your reply. I applied it and got some interesting/ slightly confusing results. I've edited the question above. $\endgroup$ – Danyal Andriano Jan 7 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.