# Training multi-label classifier with low quality training set

So I'm creating a topics classifier where a document may be tagged for several different topics, let's say - A, B while actually the document belongs to A, B and C. In the training stage I want the classifier to learn that the document belongs to A and B but I'm not sure about class C so I don't want it to learn that the document doesn't\does belong to class C. Any ideas on how to implement such thing?

I thought about adding weights to the output labels (low weight means that there's no way the document belongs to the aspect, high weight means that the document belongs to the aspect for sure and mid weight means I'm not sure (so the penalty in this case will be lower).

• I didn't get you. If the document belongs to C, why wouldn't you want to tag it with C during training? – Hima Varsha Aug 1 '16 at 7:09
• If the document belongs to C I want it to be tagged as such but what about D, E and F? I don't know if the document belongs to D or doesn't belong to it. If I'll choose to tag the document to D (meaning that the output of class D will be 1) than it will be wrong, if I'll choose to tag the document as if it doesn't belong to D (meaning of class D will be 0) it will wrong as well. I'm not sure about class D so in such case I don't want the network to change its weights as a result of class D. – Lior Magen Aug 2 '16 at 8:03
• Did I made myself clear now? – Lior Magen Aug 2 '16 at 8:04
• Are you referring to multilabel classification? – Hima Varsha Aug 2 '16 at 8:44
• Yes, just like in the title – Lior Magen Aug 2 '16 at 8:57

You almost solved the problem in the last paragraph. Expressed more formally, your cost function could be

$$\frac{1}{N} \sum_{i,j} c_{i,j} y_{i,j} \log x_{i,j}$$

where $i$ runs over items/documents, and $j$ runs over classes, $x$ is your prediction, $y$ is the binary label (1 if item $i$ has class $j$), and $0 < c < 1$ is your confidence. This is a simple modification of the cross entropy. When the confidence $c$ is low, the value of the prediction matters less.

Multilabel classification can seem to be a tough one in nlp. Recently there have been many techniques developed for this purpose.

ML-PA-LDA algorithm seems to be working well with multilabel approach. PA stands for presence absence where they consider the correlations because of absence of a class in a document.

You are right in last paragraph. You can use tensorflow for this purpose. Tensorflow supports having output as vector of classes.

The introductory tutorial of tensorflow for MNIST dataset uses output as vector of dimension 10 (one for each class 0..9)

https://www.tensorflow.org/tutorials/mnist/beginners/