I'm working on a project in R where I have roughly 1200 emails from a company, most of which are labeled class$_{1}$ or class$_{2}$, which are the types of requests. Roughly 1000 emails are labeled class$_{1}$, and 200 are labeled class$_{2}$.My goal is to used supervised learning to build a model that will classify new emails.
But, after a lot of pre-processing (parsing, removing stopwords, etc.), and trying typical algorithms (SVM, decision trees, etc.) on a document term matrix, my confusion matrix contained many false positives and false negatives, but only a few false negatives with SVM.
I'm wondering how could I improve my results? Do I need to use oversampling, or bi-gram feature representation? I guess the problem is that the topics of the two categories are really close.