Yes, it's possible to use this confidence data. However, I wouldn't recommend the approach you mention. Instead, let me suggest a different approach. Actually, I'll suggest two. The first one is conceptually clean; the second is probably easier to implement; and they'll probably be approximately equivalent in practice.
Adjust the loss function
You can adjust the loss function to reflect the confidence scores you have on the training data.
In particular, if you are using the cross-entropy loss, there's a particularly clean way to do this. Let me explain some background on the cross-entropy loss, then explain how. We think of the label on the training instance as a probability distribution on labels. In binary classification, such a distribution can be represented as a vector $(p_0,p_1)$ where $p_0$ represents the probability that the label is 0 and $p_1$ the probability that the label is 1. Normally, we are given "hard labels": if we know that the correct label on instance $x$ is 0, then that corresponds to the probability distribution $(1,0)$; if the correct label is 1, that's the distribution $(0,1)$. The cross-entropy loss then compares the prediction from the classifier to this distribution.
The nice thing about the cross-entropy loss is that it generates readily to comparing any two distributions. Thus, if you have a confidence $0.8$ that the correct label for instance $x$ is 0, then that corresponds to a probability distribution $(0.8,0.2)$. Now, you can compute the cross-entropy of the classifier's prediction with respect to the distribution $(0.8,0.2)$, and that is the contribution to the loss from training instance $x$. Sum this over all instances in the training set, and you get an adjusted loss function.
Now you can train a classifier by minimizing this adjusted loss function, and that will directly incorporate all of the information in your confidence scores.
Use weights
Alternatively, you can use weights to reflect the confidence information.
Some classifiers allow you to specify a weight for each instance in the training set. The idea is that a misprediction for a particular instance is penalized proportionality to its weight, so instances with a high weight are more important to get right and instances with a low weight are less important. Or, equivalently, the training procedure tries harder to avoid errors on instances with a high weight.
You can use weights to reflect confidence information. Suppose you have an instanced $x$ in the training set that you think should have label 0, with confidence $0.8$. You would add one copy of $(x,0)$ to the training set with weight $0.8$ (i.e., instance is $x$ and label is 0), and add one copy of $(x,1)$ to the training set with weight $0.2$ (i.e., instance is $x$ and label is 1). Build up the training set in this way. This doubles the size of your training set. Now train a classifier, using these weights.
For classifiers that support weights, this should be easy to implement.
One can also show that it is effective and reasonable. For instance, when using the cross-entropy loss to train a classifier, this approach using weights is equivalent to adjusting the loss function as highlighted above. So, in that particular context, the two approaches are actually equivalent.
