Classifying Email in R

Question

I'm working on a project in R where I have roughly 1200 emails from a company, most of which are labeled class$_{1}$ or class$_{2}$, which are the types of requests. Roughly 1000 emails are labeled class$_{1}$, and 200 are labeled class$_{2}$.My goal is to used supervised learning to build a model that will classify new emails.

But, after a lot of pre-processing (parsing, removing stopwords, etc.), and trying typical algorithms (SVM, decision trees, etc.) on a document term matrix, my confusion matrix contained many false positives and false negatives, but only a few false negatives with SVM.

I'm wondering how could I improve my results? Do I need to use oversampling, or bi-gram feature representation? I guess the problem is that the topics of the two categories are really close.

Answer 1

( All emails are or in french or in english )

Pre-processing methods :

• Merge " Summary " and " Content " which are the head of the email and the content
• Remove all emails adress
• Remove all " From : Someone To : someone ... subject : something "
• Remove all images included in the email
• Order the email according to their class
• Replace all french accent by no accent like é -> e ; ê -> e ...
• Put text to lower
• remove punctuation
• remove numbers
• strip white space
• remove some first name from a list and some specific words
• remove stop words in french and english
• stem document in french and english
• Remove term

--> Then Document term matrix, with TF-IDF

The two classes are from a technical support, two categories "complex" and "easy", "complex" are topics around finance ( in theory ) , "simple" problem with the software ( in theory ) but in practice they have a lot words in common. And the other classes I don't take them into account I just focus on the two for now

Actually the kind of classification algorithm I used is not so relevant because I tried with 5 algo and none gave good results

Example ( confusion matrix ) Decision tree:

  Decision tree:           

pred:

335 | 10
59 | 12

SVM:

331 | 1
83 | 1

Knn (n=10):

330 | 2

83 | 1

Naive Bayes:

1 | 83

12 | 320

Answer 2

Since you're dealing with only 2 classes you can make a commonality.cloud() from both classes (I use this function on R, I don't know about other languages).

It will show the words that are in common in class1 and class2. These words maybe not helping the algorithm to distinguish the classes so you can remove these words and make some tests.

Answer 3

You might want to try a Bayesian classifier instead of SVM or Decision Trees. Your problem is a generalization of the spam filter problem, i.e. classifying emails into 'Spam' (class 1) or 'Not Spam' (class 2). Most Spam filters use Bayesian classification instead of the more common supervised learning method.

Answer 4

You could study the features: I noticed you removed the numbers. It could be that you did that because they are not likely to collide, but you could also solve that by introducing word classes (@number@ or @big_number@, etc).

You could try to use the word classes on other word groups too. If that works it might indicate that your set suffers from sparsity (which wouldnt surprise me). You could use a feature selector for that purpose, for instance by ranking your dimensions on information gain.

More advanced ways to battle sparsity include: Rocchio's algorithm or word2vec.