Say I have a dataset of e-mails and I want go classify the newsletter and news emails. One way to do so, is to take all the emails I have, and call stuff that isn't newsletter nor news for other and train a classifier for three classes.

I have trained a binary-classifier which predicts really well newsletter or news but when I introduce other it starts to fall apart.

My first take on this was to train two classifiers; one which trains on other and predict_emails (where predict_emails are emails that are either newsletter or news), and if that classifier predicts predict_emails we then use the binary classifier to predict if it's newsletter or news.

I think the issue is that other contains everything else e.g spam, emails with cooking-recipies etc, thus when we try to find a pattern in other then there isn't one as such. Actually the only pattern there is, is that other is not news nor newsletter.

Because of that I thought that I would train a model which does the following: "Is it news letter? No. Is it news? No. Then it is something else of which I don't know" but I cannot figure a way to do that (thus my first take which kinda was something like it).

One of the other issue with having a three-class classifier is that say we get a new email with content we have never seen before then the classifier might have confidences of

$[P(\text{news}),P(\text{newsletter}),P(\text{other})] = [0.3,0.3,0.4]$

which in my case I want it to do "I'm very sure it is not news and it is not newsletter. Then we set it as other" thus it should be $[0.1,0.1,0.8]$.

I know I can just implement it with a logic saying that if there is not a confidence above 0.7, then set it as other, but I would rather make it "robust" and learn if there is a way to handle this "N-class + noise-class" problem

I thought about creating two "one-vs-rest" classifier (news vs other and newsletter vs other). If both predictions of these comes out low then we flag it as other.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.