# Classifying two classes when having noise as a third class

Say I have a dataset of e-mails and I want go classify the newsletter and news emails. One way to do so, is to take all the emails I have, and call stuff that isn't newsletter nor news for other and train a classifier for three classes.

I have trained a binary-classifier which predicts really well newsletter or news but when I introduce other it starts to fall apart.

My first take on this was to train two classifiers; one which trains on other and predict_emails (where predict_emails are emails that are either newsletter or news), and if that classifier predicts predict_emails we then use the binary classifier to predict if it's newsletter or news.

I think the issue is that other contains everything else e.g spam, emails with cooking-recipies etc, thus when we try to find a pattern in other then there isn't one as such. Actually the only pattern there is, is that other is not news nor newsletter.

Because of that I thought that I would train a model which does the following: "Is it news letter? No. Is it news? No. Then it is something else of which I don't know" but I cannot figure a way to do that (thus my first take which kinda was something like it).

One of the other issue with having a three-class classifier is that say we get a new email with content we have never seen before then the classifier might have confidences of

$$[P(\text{news}),P(\text{newsletter}),P(\text{other})] = [0.3,0.3,0.4]$$

which in my case I want it to do "I'm very sure it is not news and it is not newsletter. Then we set it as other" thus it should be $$[0.1,0.1,0.8]$$.

I know I can just implement it with a logic saying that if there is not a confidence above 0.7, then set it as other, but I would rather make it "robust" and learn if there is a way to handle this "N-class + noise-class" problem

I thought about creating two "one-vs-rest" classifier (news vs other and newsletter vs other). If both predictions of these comes out low then we flag it as other.