The training set is exposed to the labels, but what if a portion of these labels were incorrectly labeled by the curator? Is there a way of searching for these training set examples?
3 Answers
$\begingroup$
$\endgroup$
A good method for identifying mislabeled data is Confident Learning. It can use predictions from any trained classifier to automatically identify which data is incorrectly labeled. Since Confident Learning directly estimates which datapoints have label errors, it can also estimate what portion of these labels were incorrectly labeled by the curator.
This approach can work for most types of classification data (image, text, tabular, audio, etc) and both binary or multi-class applications. This method was previously used to discover tons of label errors in many major ML benchmark datasets.
Intuitively, a baseline solution could be flagging any example where the classifier's prediction differs from the given label. However this baseline performs poorly if the classifier makes mistakes (something typically inevitable in practice). Confident Learning also accounts for the classifier's confidence-level in each prediction and its propensity to predict certain classes (eg. some classifiers may incorrectly predict class A overly much due to a training shortcoming), in a theoretically principled way that ensures one can still identify most label errors even with an imperfect classifier.
$\begingroup$
$\endgroup$
Ideally the dataset, or at least a sample, would be annotated by several different annotators. There are several advantages to this:
- The inter-annotator agreement can be calculated. It provides some insight about how easy/objective the task is: a low value means that the task is subjective and/or difficult, so it's unlikely that an automatic system would perform very well.
- It can be used to assess the difficulty/ambiguity of an instances, since an easy instance should almost always be annotated the same way by all the annotators. Depending on the task, it might be relevant to remove ambiguous instances (not always, sometimes the difficulty is part of the task).
- It can be used to correct or check occasional errors: if an instance is annotated A by a strong majority and annotated B by a single annotator, A is likely the correct label.
- Similarly, it can be used to detect whether an annotator deviates frequently from the norm (majority of annotators). In this case this annotator's labels might be excluded.
If this is not possible, the next best thing is to train/test multiple models (typically with cross-validation) and use their predictions as if they were from different annotators. The models should be selected to be as diverse as possible (this is similar to ensemble learning).
$\begingroup$
$\endgroup$
If you have a set rules that you can set up to detect those training examples or perhaps you can build a ML model to predict which datapoint is mislabeled. A binary classification on top of a binary classification. Should be doable if you have enough data.
