2
$\begingroup$

I am looking for a specific deep learning method that can train a neural network model with both clean and noisy labels.

More precisely, I would like this method to be able to leverage noisy data as well, for instance by not fully "trusting" noisy data, or weighting samples, or deciding whether to use a specific sample at all for learning. But primarily, I am looking for inspiration.

Details:

  • My task is sequence-to-sequence NLP,
  • I have both clean pairs of sequences of (clean input, clean output) and noisy ones (noisy_input, noisy_output),
  • I know for certain which samples in my data are noisy, and if possible, I would like the desired method to make use of this information

I am very glad to give more information about my use case if needed.

Edit: Noisy vs. negative examples

First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples.

My view is that the data I have are noisy examples, but not "negative". Using an example from machine translation from German to English:

clean (equivalent meaning)

DE Wenn es um die Medien geht, lebt Amerika in einem Paralleluniversum.
EN Regarding media, the US are living in a parallel universe.

noisy (meaning overlap)

DE Wenn es um die Medien geht, lebt Amerika in einem Paralleluniversum.
EN Regarding media, the US are weird.

negative (unrelated)

DE Wenn es um die Medien geht, lebt Amerika in einem Paralleluniversum.
EN Is Math related to science?
$\endgroup$
2
  • 1
    $\begingroup$ There's a nice paper recently, Self Training With Noisy Labels, it might help. (It's for images though but idea is general) $\endgroup$
    – Aditya
    Commented Feb 17, 2020 at 3:51
  • $\begingroup$ @Aditya Thanks, will have a look! $\endgroup$ Commented Feb 17, 2020 at 7:36

2 Answers 2

3
$\begingroup$

First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples. In my opinion "noisy" is when positive and negative cases are mixed together in a way that makes it difficult (or impossible) to distinguish between them. I think this matters because you're more likely to find similar use cases and relevant methods using this terminology.

I don't have a precise method to suggest but I would check the state of the art in machine translation: it's also a sequence-to-sequence task in which there are potential positive/negative cases. In particular there has been some work done in MT quality estimation, where the goal is to predict the quality of a translation for a sentence. This might be related because it's about labeling or quantifying how good a translation is, and I would assume that there are works which re-use labelled/scored translations (including potentially wrong ones) in order to obtain a better model. Unfortunately I don't have any pointers since I haven't followed the field recently.

$\endgroup$
3
  • $\begingroup$ Thanks for your answer! I do not think "negative examples" is a good term, since the ones I have are not entirely wrong, just partially. The labels can be partially wrong since they are sequences of words. Will edit my question to clarify. My apologies if I misinterpret the meaning of "negative example". $\endgroup$ Commented Feb 16, 2020 at 19:29
  • 2
    $\begingroup$ @MathiasMüller ok I see it's not binary, so maybe you're right that the positive/negative terminology doesn't work. I'm still a bit skeptical about "noisy", but I don't have any better word. On the technical side this shares a lot of similarities with MT Quality Estimation: the "quality" can either be represented as a numerical score or with ordered classes. I vaguely remember works where the predicted "quality" was used to re-train a model somehow. $\endgroup$
    – Erwan
    Commented Feb 16, 2020 at 23:36
  • $\begingroup$ Thanks Erwan! I'm doing research in MT at the moment, so I'm aware of quality estimation methods in general - but will have another look. What I have in mind though is a generic method to train a neural network that takes into account label quality. $\endgroup$ Commented Feb 17, 2020 at 7:42
2
$\begingroup$

There is a python package created exactly for this purpose of finding label errors and training ML models robustly and reliably even when your data has issues or you have noisy labels: https://github.com/cleanlab/cleanlab -- it works for any dataset you can train a classifier on and for most data formats, ML and deep learning frameworks, and data modalities, e.g. image, text, tabular, and audio data. I am an author on this package.

Find label issues in 1 line of code

from cleanlab.classification import CleanLearning
from cleanlab.filter import find_label_issues

# Option 1 - works with sklearn-compatible models - just input the data and labels ツ
label_issues_info = CleanLearning(clf=sklearn_compatible_model).find_label_issues(data, labels)

# Option 2 - works with ANY ML model - just input the model's predicted probabilities
ordered_label_issues = find_label_issues(
    labels=labels,
    pred_probs=pred_probs,  # out-of-sample predicted probabilities from any model
    return_indices_ranked_by='self_confidence',
)

Train a model as if the dataset did not have errors -- 3 lines of code

from sklearn.linear_model import LogisticRegression
from cleanlab.classification import CleanLearning

cl = CleanLearning(clf=LogisticRegression())  # any sklearn-compatible classifier
cl.fit(train_data, labels)

# Estimate the predictions you would have gotten if you trained without mislabeled data.
predictions = cl.predict(test_data)
$\endgroup$
4
  • $\begingroup$ Thanks for the pointers, +1 ! I had a look at the library, and I think it only supports classification tasks, not sequence generation. $\endgroup$ Commented Nov 7, 2022 at 15:23
  • $\begingroup$ I think that if you give an answer like this you should disclose that you are the CEO of the company behind the library. $\endgroup$ Commented Nov 8, 2022 at 8:02
  • 1
    $\begingroup$ Happy to share a full disclosure: I wrote the cleanlab open-source package as a grad student during my 8 years at MIT. It wasn't created by a company. After I finished my PhD, I co-founded Cleanlab and raised funds so we can keep releasing new algorithms in cleanlab open-source for free for the thousands of ML engineers and data scientists who started using it while I wrote it in grad school. This is an open-source package, not a paid product. We also have a paid product called Cleanlab Studio that does more than the open-source package, which I intentionally left out of my answer. $\endgroup$ Commented Nov 8, 2022 at 20:14
  • $\begingroup$ As mentioned in my post, cleanlab supports any dataset that you can train a classifier on. if you can formulate your sequence generation as a classification task (you can for token classification and many sequence tasks -- not sure about yours, the classes = the vocabulary) then you can use cleanlab at each prediction step and obtain predicted probabilities and estimate which of your next words (tags) may be incorrect). Token classifcation: docs.cleanlab.ai/stable/tutorials/token_classification.html $\endgroup$ Commented Nov 8, 2022 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.