2
$\begingroup$

In the general text classification problem, training a machine learning model to detect if a text belongs to one of N number of classes always yields a value in N. Even if the text that was passed to the model is very far outside any of the N classes (like say you trained it on categories for movie genres, but someone passed a recipe to the classifier), it will always choose one of N as the output even though the text passed to it doesn't belong to any of the classes.

Is there a method to have an additional class for 'unknown' so that if the model output is one of the N types but matches it with very low probability then assign it to the default or unknown class?

EDIT: we are using LinearSVC from sci kit learn

$\endgroup$
2
$\begingroup$

Depends on the model but this would be a post-processing move.

E.g. in a recent problem I trained a random forest classifier and then classified any data point where the probability of the likeliest class was less than 50% as unknown.

| improve this answer | |
$\endgroup$
1
$\begingroup$

just use predict_proba and in the end if p is lower than some threshold (for 10 classes all p will be ~.1 and best could have 0.2 so if it's to low for you) you will just change predition from "class1" to "unknown", it's impossible to add it to model

| improve this answer | |
$\endgroup$
  • $\begingroup$ I was getting this when I tried AttributeError: 'LinearSVC' object has no attribute 'predict_proba' so I think the model was trained with probability=False in which case I was reading I could use decision_function instead to get the confidence for each class? $\endgroup$ – sjs Sep 23 '19 at 16:46
  • $\begingroup$ @sjs to decision_function output you should apply scipy.special.softmax to get probabilities for each class, because you will just get distance from decision boundary (with sign) $\endgroup$ – quester Sep 24 '19 at 13:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.