2
$\begingroup$

I have a dataset with around 1% of mislabeled data, it is a multi label problem and i want to find a way to correct those incorrect labels.

Assuming that the amount of mislabeled data is low i divided the dataset in Train/Test and trained a classifier taking care that the classifier does not overfit.

After that i know that the accuracy on the Test set is as high as possible i evaluated the whole dataset using the classifier and the result is a new set of labels which i assume that are the corrected labels.

Is this the correct approach to solve a problem like this?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

It think it's a reasonable approach, but currently it seems that you have no way to check whether the new labels are correct or not. I think you should at least check that the new labels don't introduce more errors than they solve.

Ideally you would re-annotate a random sample of instances, keeping both the old (possibly erroneous) labels and the new ones. Then you can use this sample as a test set and evaluate the two following points:

  • most/all instances for which the new label is the same as the old label should be predicted with this label (otherwise it means your method changes correct labels)
  • most instances for which the new label is different from the old label should be predicted with the new label (otherwise it means your method doesn't fix the wrong labels)

The problem with this approach is that you need to annotate a large sample, since you need a reasonable number of wrong labels which are only present in 1% of the data.

If re-annotating a large sample is not possible, you could try a kind of boostrapping approach: run your method, then take a sample of instances which are predicted as different from the old label. Among these labels changes, count how many are correct. This approach requires less manual annotation effort since you don't need a large random sample, however it would miss the cases of wrong label which is not changed by the classifier.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I like your answer because it goes in the direction that i was thinking as a validation for this method. Let me see if i get what you are trying to say. For example lets assume that we have 1000 samples. I should pick lets say 50 and change the label. Then after training the algorithm i should check how the 50 samples get labeled, the algorithm will be fine if from the 1000 samples, about 50 get their label changed and with the value of the previous label before changing. Is this what you mean? Thanks! $\endgroup$
    – alpuy
    Commented Nov 13, 2020 at 13:16
  • $\begingroup$ @alpuy I'm not sure I understand your solution but I don't think that's the same as what I proposed. In my first option you would need to manually re-annotate at least around 1000 instances, because you need to see what happens with at least 10-20 cases of errors and 1% of 1000 = 10. This option lets you analyze all the possible cases after training/testing. In my second option you just run the training/testing using existing labels, then after that you analyze only the cases where the predicted label is different from the original label. If the system works well, you should have only ... $\endgroup$
    – Erwan
    Commented Nov 14, 2020 at 12:14
  • $\begingroup$ ... around 1% of instances which have been changed. In this case the manual analysis is only to check how many of these instances predicted with a different label were actually errors in the original label: if for instance 90% of these instances were errors which are corrected by the new predicted label, that's good. But if more than 50% of these instances were originally correct, then the automatic re-annotation does more harm than good. $\endgroup$
    – Erwan
    Commented Nov 14, 2020 at 12:18
  • $\begingroup$ By re-annotate you mean check manually every one of the 1000 instances and correct the ones that are wrong? So we will have 1000 instances with 100% correct labels? $\endgroup$
    – alpuy
    Commented Nov 16, 2020 at 1:06
  • $\begingroup$ Yep! That's the most costly but also safest option, since it lets you evaluate the correction method on a random sample. $\endgroup$
    – Erwan
    Commented Nov 16, 2020 at 1:14
0
$\begingroup$

In case it's useful, I recently published a paper on this very topic:

Identifying Incorrect Annotations in Multi-Label Classification Data

TLDR: what you are doing is not ideal because your model typically will not produce perfect predictions when trained with limited (and noisily-labeled data). Instead you can use the multi-label extension of Confident Learning we developed in this paper to find which examples have label errors. Intuitively, this method accounts for your model's confidence as well as its propensity to predict particular classes over others (eg. perhaps your model likes to erroneously predict class A overly much due to some training shortcoming), such that you can better identify which examples have label errors even using an imperfect model.

Once you've identified which examples have label errors, you can either manually correct their labels, or delete these examples from your dataset if you are not sure what the corrected labels should be.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.