Python Text Classification - Data that does not fit into any category

Question

I am having a lot of trouble finding any kind of answers to this problem i am facing.

I have a few text classifiers that i am testing out, and they work well for data that does fit into any predefined category, but if I input, lets just say "fhjakdlfsah", it will still assign it to some category because i guess the predict_proba functionality has to add up to 1 for all of the categories.

Is there something i am missing here? I am having such a hard time finding a solution to this, and I would imagine it is a very common thing to deal with. Right now i am working with gradientboosting from sklearn, and tried wrapping it in a onevsrestclassifier as suggested by others, but it is still having the same thing where all probabilities are adding up to one, and it is getting assigned the highest probability

Basically I am looking for a solution that can say either, yes this fits into one of these categories, or no, this does not fit into any of these categories.

Any help would be greatly appreciated as I am getting quite stuck here

Thanks!

Answer 1

The main assumption in supervised learning is that the training set is a representative sample of the space of all possible instances for the problem.

This is why in the regular classification setting there is simply no way to consider a "not in any class" option: what the model learns is to distinguish between the classes seen in the training set, nothing else. In the standard one-vs-rest multi-class setting It's important to understand that the model doesn't predict whether an instance is likely to belong to "class A" in general, it only predicts whether the instance is more likely to belong to class A compared to any other known class. This is a common misunderstanding, but one-vs-rest is not an answer to the problem of "not in any class".

There are options to deal with this problem, but one must design the system so that the model can deal with such cases. In general the problem is actually open-set classification: in regular classification the set of possible classes is closed, as mentioned above. The open-set classification setting is much less standard and usually harder, since the model has to make a prediction about something it does not know from its training data. Here are a few options:

• Treat the problem as an outlier detection problem: before running the regular classifier, eliminate abnormal instances which don't look like the regular training set cases.
• Multi-label classification: in multi-label classification an instance may belong to zero, one or several classes. A binary classifier is trained for every class, and an instance may belong to none of the classes. This is more flexible than multi-class classification, but it's still closed-set classification: for example the model is supposed to have seen examples in the training set which don't belong to any class.
• One-class classification is a type of classification where the model learns to identify a single class "in general", not by contrast to any other class. This is proper open-set classification: if a model is trained for every known class and every model predicts the instance as negative, then the instance doesn't belong to any known class.

Answer 2

There are cutting edge research available on open class classification, however a workaround which is commonly used in production is to use the probability as prediction confidence and only predict class where model is confident. For example when you predict for 'fhjakdlfsah' it should predict the class with a low probability compared to a correct prediction. So you would always have some identified subset of data which your model cannot predict, you may call it fallout/fallback as commonly termed in chatbots. In chatbots the fallback is generally handled by human agents.

Key question is what should be the cut-off of probability above which predictions are generally correct? Should it be 0.7/ 0.8/ anything else? Answer to that would come from your specific use case. On a couple of test datasets you need to predict class and probability. Then plot how fallout% and accuracy of non fallout portion change with different choices of probability cutoff. Choose a cutoff which suits your accuracy need as well as manageable volume of fallout.

You need to be cautious about probability scale in case your model is not well calibrated. Probabilities may generally be on the higher side. But still the above explained plot should give you an answer.