1
$\begingroup$

Here is some background on the problem. My aim is to classify text into some categories. I would like to get only good quality predictions from the model. If the model is not confident, I would like to classify the text manually.

Let's consider the example provided in http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html so that it's reproducible. In the following example, a classification model is trained and is fitted for test documents. One of the test document is - 'what the heck is this?'. I understand that model is returning the class which has highest probability. However, when the model is unsure, I would like to label the text as 'unable to classify'

from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB().fit(X_train_tfidf, twenty_train.target)

docs_new = ['God is love', 'OpenGL on the GPU is fast', 'what the heck is this?']
X_new_counts = count_vect.transform(docs_new)
X_new_tfidf = tfidf_transformer.transform(X_new_counts)

predicted = clf.predict(X_new_tfidf)

for doc, category in zip(docs_new, predicted):
    print('%r => %s' % (doc, twenty_train.target_names[category]))

Output

'God is love' => soc.religion.christian
'OpenGL on the GPU is fast' => comp.graphics
'what the heck is this?' => soc.religion.christian

Predicting the probabilities

Here are the probabilities of prediction. Documents 1 and 2 have somewhat clear winner. However, the third document doesn't have one. I have about 100 classes and I would be hesitant to set a manual threshold.

clf.predict_proba(X_new_tfidf)
array([[ 0.16297502,  0.03828016,  0.03737814,  0.76136668],
       [ 0.16387956,  0.36874738,  0.2364763 ,  0.23089675],
       [ 0.28288106,  0.17035852,  0.2484853 ,  0.29827513]])
Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ You need to define what you mean low quality prediction. If it's that no prediction is above a certain threshold then that's rather domain specific and up to how much risk and reward the end user is comfortable with. If it's that two classes probabilities are too close together, then you need to define a metric and determine what the threshold should be. Either way the implementation is simple enough once you clearly define what you want. $\endgroup$ – Tophat Dec 19 '17 at 18:58
1
$\begingroup$

An alternative to thresholding the classification probability would be to set a threshold on the ratio between the highest reported probability and the second highest reported probability. For example, a threshold of 2 would be interpretable as: "Only retain classifications where the likelihood of the class assignment is at least twice as likely as the next most likely class."

Improve this answer
$\endgroup$
0
$\begingroup$

Model confidence is domain specific.

Thus you can manually set a threshold. For example, if P < .65 then manually classification.

Or you can train a second machine learning system to learn what level of confidence is required for the specific task.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.