0
$\begingroup$

In an image classification task, I know there are mislabeled data. should I remove/correct them in all training / validation / test set ?

I saw this article https://arxiv.org/pdf/2103.14749.pdf but I am not sure if I understood the result correctly - does adding mislabel improve the performance ?!

$\endgroup$

2 Answers 2

0
$\begingroup$

I'd say yes : if you have mislabeled data in train, the model won't learn correctly, and if you have them in test/val, your test results won't represent the actual model results.

Since that's directly the label, that's ok to correct/remove them, since you'd never have the label in real new cases. Moreover, if you encounter the same problem with a variable, it's more difficult to deal with, since the variable may also have misvalues in the future application of your model.

$\endgroup$
2
$\begingroup$

I’m the lead author of that paper :)

Mislabeled data can detrimentally effect the performance of ML models. The primary takeaway of that paper is that you need to use accurately labeled test sets when benchmarking. You also should train your models on accurately labeled data.

The easiest (one-line-of-code) way to find mislabeled data is via the cleanlab data-centric AI python package: https://github.com/cleanlab/cleanlab which works for any dataset you can train a classifier on for most data formats, ML and deep learning frameworks, and data modalities, e.g. image, text, tabular, and audio data.

Find label issues in 1 line of code

from cleanlab.classification import CleanLearning
from cleanlab.filter import find_label_issues

# Option 1 - works with sklearn-compatible models - just input the data and labels ツ
label_issues_info = CleanLearning(clf=sklearn_compatible_model).find_label_issues(data, labels)

# Option 2 - works with ANY ML model - just input the model's predicted probabilities
ordered_label_issues = find_label_issues(
    labels=labels,
    pred_probs=pred_probs,  # out-of-sample predicted probabilities from any model
    return_indices_ranked_by='self_confidence',
)

Train a model as if the dataset did not have errors -- 3 lines of code

from sklearn.linear_model import LogisticRegression
from cleanlab.classification import CleanLearning

cl = CleanLearning(clf=LogisticRegression())  # any sklearn-compatible classifier
cl.fit(train_data, labels)

# Estimate the predictions you would have gotten if you trained without mislabeled data.
predictions = cl.predict(test_data)

Documentation and runnable tutorials for cleanlab: https://docs.cleanlab.ai/

The above approaches find issues. To correct label issues in your dataset...

The above approaches, just find issues and train without them. To correct the labels or train a model on a corrected dataset (that still includes all the errors, but now corrected with the right label), there is a no-code tool for that called Cleanlab Studio (https://cleanlab.ai/studio) for which I am also an author.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.