Eliminate low quality predictions in a classification task

Question

Here is some background on the problem. My aim is to classify text into some categories. I would like to get only good quality predictions from the model. If the model is not confident, I would like to classify the text manually.

Let's consider the example provided in http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html so that it's reproducible. In the following example, a classification model is trained and is fitted for test documents. One of the test document is - 'what the heck is this?'. I understand that model is returning the class which has highest probability. However, when the model is unsure, I would like to label the text as 'unable to classify'

from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB().fit(X_train_tfidf, twenty_train.target)

docs_new = ['God is love', 'OpenGL on the GPU is fast', 'what the heck is this?']
X_new_counts = count_vect.transform(docs_new)
X_new_tfidf = tfidf_transformer.transform(X_new_counts)

predicted = clf.predict(X_new_tfidf)

for doc, category in zip(docs_new, predicted):
    print('%r => %s' % (doc, twenty_train.target_names[category]))

Output

'God is love' => soc.religion.christian
'OpenGL on the GPU is fast' => comp.graphics
'what the heck is this?' => soc.religion.christian

Predicting the probabilities

Here are the probabilities of prediction. Documents 1 and 2 have somewhat clear winner. However, the third document doesn't have one. I have about 100 classes and I would be hesitant to set a manual threshold.

clf.predict_proba(X_new_tfidf)
array([[ 0.16297502,  0.03828016,  0.03737814,  0.76136668],
       [ 0.16387956,  0.36874738,  0.2364763 ,  0.23089675],
       [ 0.28288106,  0.17035852,  0.2484853 ,  0.29827513]])

Answer 1

An alternative to thresholding the classification probability would be to set a threshold on the ratio between the highest reported probability and the second highest reported probability. For example, a threshold of 2 would be interpretable as: "Only retain classifications where the likelihood of the class assignment is at least twice as likely as the next most likely class."

Answer 2

Model confidence is domain specific.

Thus you can manually set a threshold. For example, if P < .65 then manually classification.

Or you can train a second machine learning system to learn what level of confidence is required for the specific task.