I am working to build a text classifier using a Boosting method from sklearn. It is performing quite well, at around 97% accuracy on my test data. However, the problem I am seeing is that if I input text that clearly does not fall into a predefined category, it will randomly assign it to a certain classification with a high probability score

For example:

X_train = df_train.text
X_test = df_test.text
y_train = df_train.label
y_test = df_test.label

boosting = Pipeline([('vect', CountVectorizer()),
('tfidf', TfidfTransformer()),
boosting = boosting.fit(df_train.text, df_train.label)


list of categories -> {'A': fruit, 'B': animal, 'C': car, 'D': person, 'E': dessert, 'F': place}

docs = ['this is not category']

boosting.predict_proba(docs).tolist()


Output:

[[0.0016872033185414193,
0.9915417761339475,
0.0016865302624752719,
0.0016921961567993337,
0.0016974399174602914,
0.001694854210776211]]


You can see that the second category is receiving a .99 probability when it is clearly not fitting into any of the options. Regardless of what I put through it, could be "fdahsjfkasl" it will return the same probability score for that second category

The model works so well for text that could logically fit into a category (not only performing well on test data, but also on new/random text too), but i need a way to handle text that does not, so that it can be labeled "Not a category" or something like it.

Does anyone have any suggestions?

This will always happen since in multi-output classification (OnevsOne or OnevsRest) classes are mutually exclusive i.e probabilities will always sum up to one.

If you want independent probabilities you could try rewrite your target so that this is a multi-output classification (not a multi class) problem

toy example with iris data:

from sklearn.datasets import load_iris
from sklearn.multioutput import MultiOutputClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder

X_train, X_test, y_train, y_test = train_test_split(X,y,random_state= 42, stratify= y)

model = RandomForestClassifier(random_state= 42).fit(X_train,y_train)



Output:

Rewriting in a multi-output classification:

y_train = y_train.reshape(-1, 1)
y_test = y_test.reshape(-1, 1)

encoder = OneHotEncoder(handle_unknown= "ignore").fit(y_train)

y_train_multiclass = np.array(encoder.transform(y_train).todense())
y_test_multiclass = np.array(encoder.transform(y_test).todense())

model = MultiOutputClassifier(estimator= RandomForestClassifier(random_state= 42)).fit(X_train,y_train_multiclass)

pred = np.array(model.predict_proba(X_test))

pred[:,:,1].shape

prob0 = pred[:,:, 0].T
prob1 = pred[:,:, 1].T



Output:

Yo can see that in this approach the probabilities might not sum up to one, so it may happen that if you pass a document that does not belong to any category I should return 0 for all the classes

Other option would be to use a neural net model and changing the utput via the activation function of the output layer as mention on this answer:

Should estimated probabilities from multi class classification sum to 1

• Thanks for the explanation, can you help me understand why you reshape y_train and y_test to (-1,1)? and do I need to also reshape X_train and X_test?
– Jim
May 12 at 17:16
• y is a numpy array with shape (n,) OneHotEncoder needs an array with one column i.e shape (n,1) May 12 at 23:38