9
$\begingroup$

I have training data that is labelled with binary values. I also have collected the confidence of each of these labels i.e. 0.8 confidence would mean that 80% of the human labellers agree on that label.

Is it possible to use this confidence data to improve the accuracy of my classifier?

Would the following work?

  • 1a) If the label is 0 and the confidence data for that label is 0.8 then give the training data a new label of 0.2

  • 1b) If the label is 1 and the confidence data for that label is 0.8 then give the training data a new label of 0.8

  • 2) Calculate the new label using this method for every entry in the training set

  • 3) Treat the problem as a regression problem (where the labels range between 0 and 1)

  • 4) Classify unlabelled data based on whether the new labels are above or below a specific value. i.e Give all predicted labels class 0 if below X and class 1 if above X.

I am currently using a SVM with an RBF kernel for my model.

Thanks in advance!

$\endgroup$
  • $\begingroup$ Classifier for uncertain class labels $\endgroup$ – Emre May 24 '17 at 16:44
  • $\begingroup$ So for now you are using the binary label ? I don't know why it wouldn't work. Test it. Split your data into a training test and a validation test and see if you increase your prediction. $\endgroup$ – el Josso May 25 '17 at 8:09
4
$\begingroup$

Yes, it's possible to use this confidence data. However, I wouldn't recommend the approach you mention. Instead, let me suggest a different approach. Actually, I'll suggest two. The first one is conceptually clean; the second is probably easier to implement; and they'll probably be approximately equivalent in practice.

Adjust the loss function

You can adjust the loss function to reflect the confidence scores you have on the training data.

In particular, if you are using the cross-entropy loss, there's a particularly clean way to do this. Let me explain some background on the cross-entropy loss, then explain how. We think of the label on the training instance as a probability distribution on labels. In binary classification, such a distribution can be represented as a vector $(p_0,p_1)$ where $p_0$ represents the probability that the label is 0 and $p_1$ the probability that the label is 1. Normally, we are given "hard labels": if we know that the correct label on instance $x$ is 0, then that corresponds to the probability distribution $(1,0)$; if the correct label is 1, that's the distribution $(0,1)$. The cross-entropy loss then compares the prediction from the classifier to this distribution.

The nice thing about the cross-entropy loss is that it generates readily to comparing any two distributions. Thus, if you have a confidence $0.8$ that the correct label for instance $x$ is 0, then that corresponds to a probability distribution $(0.8,0.2)$. Now, you can compute the cross-entropy of the classifier's prediction with respect to the distribution $(0.8,0.2)$, and that is the contribution to the loss from training instance $x$. Sum this over all instances in the training set, and you get an adjusted loss function.

Now you can train a classifier by minimizing this adjusted loss function, and that will directly incorporate all of the information in your confidence scores.

Use weights

Alternatively, you can use weights to reflect the confidence information.

Some classifiers allow you to specify a weight for each instance in the training set. The idea is that a misprediction for a particular instance is penalized proportionality to its weight, so instances with a high weight are more important to get right and instances with a low weight are less important. Or, equivalently, the training procedure tries harder to avoid errors on instances with a high weight.

You can use weights to reflect confidence information. Suppose you have an instanced $x$ in the training set that you think should have label 0, with confidence $0.8$. You would add one copy of $(x,0)$ to the training set with weight $0.8$ (i.e., instance is $x$ and label is 0), and add one copy of $(x,1)$ to the training set with weight $0.2$ (i.e., instance is $x$ and label is 1). Build up the training set in this way. This doubles the size of your training set. Now train a classifier, using these weights.

For classifiers that support weights, this should be easy to implement.

One can also show that it is effective and reasonable. For instance, when using the cross-entropy loss to train a classifier, this approach using weights is equivalent to adjusting the loss function as highlighted above. So, in that particular context, the two approaches are actually equivalent.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.