GradientBoostingRegressor Text Classifier

Question

I am working to build a text classifier using a Boosting method from sklearn. It is performing quite well, at around 97% accuracy on my test data. However, the problem I am seeing is that if I input text that clearly does not fall into a predefined category, it will randomly assign it to a certain classification with a high probability score

For example:

X_train = df_train.text
X_test = df_test.text  
y_train = df_train.label
y_test = df_test.label

boosting = Pipeline([('vect', CountVectorizer()),
                  ('tfidf', TfidfTransformer()),
                  ('boosting', GradientBoostingClassifier()),])
boosting = boosting.fit(df_train.text, df_train.label)

list of categories -> {'A': fruit, 'B': animal, 'C': car, 'D': person, 'E': dessert, 'F': place}

docs = ['this is not category']

boosting.predict_proba(docs).tolist()

Output:

[[0.0016872033185414193,
  0.9915417761339475,
  0.0016865302624752719,
  0.0016921961567993337,
  0.0016974399174602914,
  0.001694854210776211]]

You can see that the second category is receiving a .99 probability when it is clearly not fitting into any of the options. Regardless of what I put through it, could be "fdahsjfkasl" it will return the same probability score for that second category

The model works so well for text that could logically fit into a category (not only performing well on test data, but also on new/random text too), but i need a way to handle text that does not, so that it can be labeled "Not a category" or something like it.

Does anyone have any suggestions?

Answer 1

This will always happen since in multi-output classification (OnevsOne or OnevsRest) classes are mutually exclusive i.e probabilities will always sum up to one.

If you want independent probabilities you could try rewrite your target so that this is a multi-output classification (not a multi class) problem

toy example with iris data:

from sklearn.datasets import load_iris
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.multioutput import MultiOutputClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder

X, y = load_iris(return_X_y= True)

X_train, X_test, y_train, y_test = train_test_split(X,y,random_state= 42, stratify= y)

model = RandomForestClassifier(random_state= 42).fit(X_train,y_train)

pd.DataFrame(model.predict_proba(X_test)).head(5)

Output:

Rewriting in a multi-output classification:

y_train = y_train.reshape(-1, 1)
y_test = y_test.reshape(-1, 1)

encoder = OneHotEncoder(handle_unknown= "ignore").fit(y_train)

y_train_multiclass = np.array(encoder.transform(y_train).todense())
y_test_multiclass = np.array(encoder.transform(y_test).todense())

model = MultiOutputClassifier(estimator= RandomForestClassifier(random_state= 42)).fit(X_train,y_train_multiclass)

pred = np.array(model.predict_proba(X_test))

pred[:,:,1].shape

prob0 = pred[:,:, 0].T
prob1 = pred[:,:, 1].T

pd.DataFrame(prob0).head(5)

Output:

Yo can see that in this approach the probabilities might not sum up to one, so it may happen that if you pass a document that does not belong to any category I should return 0 for all the classes

Other option would be to use a neural net model and changing the utput via the activation function of the output layer as mention on this answer:

Should estimated probabilities from multi class classification sum to 1