Multi-label text classification with minimum confidence threshold

Question

Building out a system that tries to apply zero or more predefined labels to text.

For each label, we've:

• built out a reasonably good vocabulary of high-value words/features
• developed a corpus containing thousands of labeled entries
• trained a NaiveBayesClassifier for each topic that does a good job of classifying valid vs noisy content

The problem seems to be that the individual classifier is great at differentiating between valid & noisy content WITHIN a topic:

• "the green energy bill will revolutionize..." (green = "green energy")
• "the green bay packers went on to lose their..." (green != "green energy")

...but when classifying content that shouldn't match ANY topic it has a very high rate of false positives. There's no "everything else" label!

tl;dr it's good at subtle, in-topic differentiation, but terrible at broad topic labeling

Are there any algorithms that help you classify into N categories, but allow for "everything else" which might not fit into ANY of the categories?

Answer 1

Looks like the common approaches to multi-class classification actually solve this challenge.

Building individual Naive Bayes Classifiers with only the training data for a single label is insufficient - we must also the include data from other labels as "everything else".

See Text-Classification-Problem, what is the right approach?