I have a corpus of text documents, some of which are labelled by analysts with label L. I am using this data to train an SVM for predicting if a new document should have label L.
So far it's straight-forward, but there is an issue: The analysts have not evaluated all documents in the training set, so there are in fact three classes of documents:
- Documents labeled L
- Documents the analysts have looked at, and chosen not label L (so you could say they're labelled not-L)
- Documents the analysts have not looked at.
Unfortunately, at training time, I have no way to separate documents in 2. and 3, or not-L and unlabelled documents. I believe this is a problem, because a not-L label gives information to the SVM, but an unlabelled document is more "neutral".
How can I estimate the impact of this issue on predicting if a new document should have label L?