I have a dataset with around 1% of mislabeled data, it is a multi label problem and i want to find a way to correct those incorrect labels.

Assuming that the amount of mislabeled data is low i divided the dataset in Train/Test and trained a classifier taking care that the classifier does not overfit.

After that i know that the accuracy on the Test set is as high as possible i evaluated the whole dataset using the classifier and the result is a new set of labels which i assume that are the corrected labels.

Is this the correct approach to solve a problem like this?


It think it's a reasonable approach, but currently it seems that you have no way to check whether the new labels are correct or not. I think you should at least check that the new labels don't introduce more errors than they solve.

Ideally you would re-annotate a random sample of instances, keeping both the old (possibly erroneous) labels and the new ones. Then you can use this sample as a test set and evaluate the two following points:

  • most/all instances for which the new label is the same as the old label should be predicted with this label (otherwise it means your method changes correct labels)
  • most instances for which the new label is different from the old label should be predicted with the new label (otherwise it means your method doesn't fix the wrong labels)

The problem with this approach is that you need to annotate a large sample, since you need a reasonable number of wrong labels which are only present in 1% of the data.

If re-annotating a large sample is not possible, you could try a kind of boostrapping approach: run your method, then take a sample of instances which are predicted as different from the old label. Among these labels changes, count how many are correct. This approach requires less manual annotation effort since you don't need a large random sample, however it would miss the cases of wrong label which is not changed by the classifier.

  • $\begingroup$ I like your answer because it goes in the direction that i was thinking as a validation for this method. Let me see if i get what you are trying to say. For example lets assume that we have 1000 samples. I should pick lets say 50 and change the label. Then after training the algorithm i should check how the 50 samples get labeled, the algorithm will be fine if from the 1000 samples, about 50 get their label changed and with the value of the previous label before changing. Is this what you mean? Thanks! $\endgroup$ – alpuy Nov 13 '20 at 13:16
  • $\begingroup$ @alpuy I'm not sure I understand your solution but I don't think that's the same as what I proposed. In my first option you would need to manually re-annotate at least around 1000 instances, because you need to see what happens with at least 10-20 cases of errors and 1% of 1000 = 10. This option lets you analyze all the possible cases after training/testing. In my second option you just run the training/testing using existing labels, then after that you analyze only the cases where the predicted label is different from the original label. If the system works well, you should have only ... $\endgroup$ – Erwan Nov 14 '20 at 12:14
  • $\begingroup$ ... around 1% of instances which have been changed. In this case the manual analysis is only to check how many of these instances predicted with a different label were actually errors in the original label: if for instance 90% of these instances were errors which are corrected by the new predicted label, that's good. But if more than 50% of these instances were originally correct, then the automatic re-annotation does more harm than good. $\endgroup$ – Erwan Nov 14 '20 at 12:18
  • $\begingroup$ By re-annotate you mean check manually every one of the 1000 instances and correct the ones that are wrong? So we will have 1000 instances with 100% correct labels? $\endgroup$ – alpuy Nov 16 '20 at 1:06
  • $\begingroup$ Yep! That's the most costly but also safest option, since it lets you evaluate the correction method on a random sample. $\endgroup$ – Erwan Nov 16 '20 at 1:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.