Building a multiclass classifier that can handle classes it has never seen?

Question

I am given a dataset that has free-form text and a category associated with it. There are 100 different categories and 3000 records for each category. The goal is build a multiclass classification model. I have created a simple neural network with 10,000 input features/words and the results were fairly good (~88%).

The issue I am facing is with an unlabeled dataset that I have, that is missing the category label. This dataset is very large, and has much more than 100 categories. I am only interested in being able to classify unlabeled data for the 100 categories that I have, but I am not sure how to approach this.

One thought I had was to build a Word Embedding model for the labeled data. This model could be used to calculate a document vector for the unlabeled data and find a similar document from the labeled dataset. This would allow me to assign labels to some of the data in the unlabeled dataset. Is there a better way to approaching this problem?

Answer 1

You basically want your model to say - I dont know.

For the test set created out of the original dataset with labels, plot a histogram of the probabilities of the predicted class. Are they generally higher than some threshold say 0.2 ? If so experiment using that as a threshold to flag items from your unlabeled dataset for a manual inspection. The hope is that the probabilities are low because the model is unsure.

The manual review of these flagged items will allow you to verify if this approach even works. If it works, then use that as a labeling strategy for the "I dont know" category and retrain along with this new label.

Answer 2

If you necessarily want to predict a category for an unlabeled product, then you can use TF-IDF Vectorization which is similar to what you are thinking of. Using cosine similarity, you can find the top 5 most similar documents and on the basis of majority of their categories, you can predict the category of the test document.

Answer 3

A common approach is to use ensemble uncertainty.

Train a few versions of your model (say 5 versions), where you initialise each training process with a different seed. For any new data point, you evaluate it using all your 5 trained models and then see how well they agree - if they don't agree well, you can say I don't know. You can measure agreement quantitatively using KL divergence between the probability distributions output by your models.