# Naives Bayes Text Classifier Confidence Score

I am experimenting with building a text classifier using Naive Bayes which has been pretty successful on my test data. One thing i am looking to incorporate is handling text that does not fit into any predefined category that I trained the model on.

Does anyone have some thoughts on how to do this? I was thinking of trying to calculate the confidence score for each document, and if < 80 % confidence, for example, it should label the data as "N/A"

This is my code so far:

df_train = pd.read_csv(__________)

text_clf = Pipeline([('vect', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf', MultinomialNB()),])
text_clf = text_clf.fit(df_train.text, df_train.label)

df['predicted'] = predicted


Like I said, it works well for documents that do fit into one of the categories, but if I have something that clearly does not fit into anything, it will still try and assign it a label, my guess is based on some kind of confidence calculation but just not sure how that works

First there is a simple theoretical reason why relying on the probability provided by a NB model is not a good idea: what NB predicts is the posterior probability, i.e. the conditional probability $$p(C_i|d)$$ for every class $$C_i$$ for a given document $$d$$. The class $$C$$ which is predicted is just the one which obtains the highest probability $$p(C_i|d)$$. The sum of all the posterior probabilities for a particular $$d$$ is 1 since it's a conditional, and this means that a class is always predicted relatively to the other classes: a high probability for $$C$$ doesn't mean that the document is very likely to be $$C$$ in general, it only means that it is much more likely to be $$C$$ than any of the other classes. If it actually belongs to another class unknown to the model, there's no way the model can represent this.