# Impact of unlabelled documents for label prediction via SVM

I have a corpus of text documents, some of which are labelled by analysts with label L. I am using this data to train an SVM for predicting if a new document should have label L.

So far it's straight-forward, but there is an issue: The analysts have not evaluated all documents in the training set, so there are in fact three classes of documents:

1. Documents labeled L
2. Documents the analysts have looked at, and chosen not label L (so you could say they're labelled not-L)
3. Documents the analysts have not looked at.

Unfortunately, at training time, I have no way to separate documents in 2. and 3, or not-L and unlabelled documents. I believe this is a problem, because a not-L label gives information to the SVM, but an unlabelled document is more "neutral".

How can I estimate the impact of this issue on predicting if a new document should have label L?

• To test the impact you could do cross validation on a test set. Even then you would only be able to test the accuracy of one class. But, yes, I would think this is an issue. Generally, SVMs are binary classifiers and your non-L will be very noisy. Have you considered a nearest neighbour or clustering to find those similar to the L records? – user13684 Nov 20 '15 at 2:09
• I had not considered clustering, no. It seems my problem is more similar to a semi-supervised learning problem than straight binary classification. Thanks for the idea. – André Risnes Nov 20 '15 at 9:51

It's a slight variation on your problem as described, but if your goal is to build a good model for predicting documents labeled $L$, I would initially formulate this as a recommender system problem, until you've reached a desirable point in the learning curve of your system. I did exactly this in a publication a few years back. In my approach, all labeled documents are used to train the SVM, while all those unlabeled are used for classifying. I use the signed margin distance of each classified document for ranking, and take the top-$n$ as the next $n$ documents the reviewers should assign a label $\in(L, L_{not})$. A side effect of this process is that you get to iteratively evaluate the performance of your classifier as you add new labeled documents to the model, which should get you what you need. If this approach sounds useful to you, I recommend looking over my paper, as I describe some performance metrics useful for this, in addition to outlining the specific procedure to use that will ensure you're not biasing your model.