I am working on a project for a company which needs to categorize customer e-mails regarding loans and insurance. The e-mails are labeled uniquely from set of 13 category labels. The number of records in the dataset are 3000. All e-mails are not in proper English or format, there are a lot of spelling mistakes and unwanted features as Indian Names, addresses etc.. The dataset is very imbalanced, the frequency of top 3 categories combines upto 2000 and there are 2 categories with frequencies ranging from 40-80. I managed to pre-process the data and remove most of the unwanted stuff and spelling mistakes.
I have created a supervised learning model using LinearSVC which gave around 78% on validation dataset, after which I used XGBoost classifier which increased the accuracy to 80%. Both of these models use Tfidf Vectors with unigrams and bigrams of the pre-processed(stop words removed and stemmed text) as input. I have tried SMOTE, TruncatedSVD and SelectKBest with chi2 but they decreased the accuracy.
In terms of Machine Learning Models I have used Multinomial Naive Bayes, Random Forests, Decision Tree Classifiers. I have even tried Feed Forward Neural Networks(MLP) and Recurrent Neural Networks for the classification but still XGBoost gives the best accuracy. I have also used Topical Modeling using LDA. I generated top words for each category and tried finding cosine similarities of each test set record with all the generated top words belonging to each category, still no improvement in accuracy.
So, my question is what kind of features should I look for? How do I extract more sophisticated features? My main objective is to increase the accuracy as the product is going to be an in-house software for the company. I am looking forward to context aware feature extraction or somehow engineer the features(add more or enhance them) to increase accuracy
PS: In the emails the customer ask for multiple actionable items, but still in the training set they are pre-classified as only one label corresponding to one of the actionable item asked in the e-mail. Plus, the data is also manually wrongly classified by the previous workers. Please suggest and approach to capture the actionable items. I Have tried using class probabilities but they don't seem like giving the right answer, plus if the person has asked for only one actionable item then also the class probabilities will indicate other. I want to capture the actionable items as per asked by User. I have tried POS-tagging for this but due to bad quality of data, it is not giving good results.
PPS: The Email data which I have is in HTML format and has threads. I tried removing previous conversation threads before training but it resulted in bad accuracy. It is very strange that the more I try to refine the data by removing unwanted contents like disclaimers, auto-response emails, replies from customer-care representatives the more the accuracy falls.
Looking forward to get some insights on how to change my approach and specifically the features.