Categorizing Customer Emails

Question

I am working on a project for a company which needs to categorize customer e-mails regarding loans and insurance. The e-mails are labeled uniquely from set of 13 category labels. The number of records in the dataset are 3000. All e-mails are not in proper English or format, there are a lot of spelling mistakes and unwanted features as Indian Names, addresses etc.. The dataset is very imbalanced, the frequency of top 3 categories combines upto 2000 and there are 2 categories with frequencies ranging from 40-80. I managed to pre-process the data and remove most of the unwanted stuff and spelling mistakes.

I have created a supervised learning model using LinearSVC which gave around 78% on validation dataset, after which I used XGBoost classifier which increased the accuracy to 80%. Both of these models use Tfidf Vectors with unigrams and bigrams of the pre-processed(stop words removed and stemmed text) as input. I have tried SMOTE, TruncatedSVD and SelectKBest with chi2 but they decreased the accuracy.

In terms of Machine Learning Models I have used Multinomial Naive Bayes, Random Forests, Decision Tree Classifiers. I have even tried Feed Forward Neural Networks(MLP) and Recurrent Neural Networks for the classification but still XGBoost gives the best accuracy. I have also used Topical Modeling using LDA. I generated top words for each category and tried finding cosine similarities of each test set record with all the generated top words belonging to each category, still no improvement in accuracy.

So, my question is what kind of features should I look for? How do I extract more sophisticated features? My main objective is to increase the accuracy as the product is going to be an in-house software for the company. I am looking forward to context aware feature extraction or somehow engineer the features(add more or enhance them) to increase accuracy

PS: In the emails the customer ask for multiple actionable items, but still in the training set they are pre-classified as only one label corresponding to one of the actionable item asked in the e-mail. Plus, the data is also manually wrongly classified by the previous workers. Please suggest and approach to capture the actionable items. I Have tried using class probabilities but they don't seem like giving the right answer, plus if the person has asked for only one actionable item then also the class probabilities will indicate other. I want to capture the actionable items as per asked by User. I have tried POS-tagging for this but due to bad quality of data, it is not giving good results.

PPS: The Email data which I have is in HTML format and has threads. I tried removing previous conversation threads before training but it resulted in bad accuracy. It is very strange that the more I try to refine the data by removing unwanted contents like disclaimers, auto-response emails, replies from customer-care representatives the more the accuracy falls.

Looking forward to get some insights on how to change my approach and specifically the features.

Thank You

Answer 1

(1) Data quality. The single best way to improve your accuracy. Garbage in garbage out. You already said your data is suspect. Some data was mis-classified; data only has single label, when multiple labels are possible. This is the biggie - this will improve your accuracy more than any other technique: improve the quality of your training data. One way to go about this. Recruit expert labellers (these could be workers in your own company who do this task, or they could be external people, e.g., mturkers who have been trained to do this labelling task). In general, the more labellers, the better. At a minimum I would have 3 labellers, label all n emails in your train/test set, using either majority label wins, or require all-three consensus depending on your confidence needs. You will want to measure your inter-rater reliability to ensure the integrity of your data using Cohen's kappa or similar. Importantly, this data will become your gold standard data. That is to say the average agreement upon labels provided by your original labellers is the ground truth to which your algorithm attempts to approach.

Obviously, this approach to data quality is costly in terms of resources (time to label all the emails; cost of hiring mturkers, etc.). But depending upon your specific needs it needn't be... basically it depends upon the needed level of "expertise" to do the task. Sometimes, it is safe to consider you (the researcher/developer) as the "expert" (e.g., you yourself said that some emails were mis-classified... how do you know that? You must have some sort of expert knowledge.). However, using a single labeller for the task makes your data less reliable due to individual processing differences introducing a bias into the data.

(2) Including stop words. Yes, you read that correct including stop words. Grammatical function words (e.g., the, and, of, etc.) can often be highly indicative of specific text types or genres. It is premature to eliminate stop words without first ensuring that they are an uninformative feature. Similarly, try including punctuation, try including capitalization features. Time and time again, I see people blindly pre-processing text without empirically determining whether these features are significant predictors of class.

(3) Capture spelling variation. As the OP stated, the original texts have many "spelling mistakes." You can improve accuracy by reducing this variance - basically map variants to their canonical form before extracting your word features (e.g., "colour" -> "color"). I often rely on external corpora to rapidly create these spelling maps.

(4) Balance your training data - The OP mentions that his classes are severely imbalanced. Try over- and/or under-sampling to balance your classes.

(5) Describe your problem. It is also important to understand what the underlying purpose of your classifier. If your classifier is intended to run on historical data, perhaps to flag incorrectly human-classified emails, then you would probably want to keep auto-response and correspondence thread info. On the other hand, if the goal is to classify new customer emails so that they get properly routed, for example, then you probably want to remove these correspondence features from your training and test data.

Answer 2

First of all, I think that your accuracy is already very high for text classification. I want to provide some ideas for additional features and approaches though.

Topic models

Topic models such as Latent Dirichlet Allocation (LDA) are quite frequently used when studying text corpora. You already wrote that you used LDA for coming up with cosine similarities. What you could also do though is to use the latent topics as additional features for individual documents, and use the probabilistic assignments as values.

Sentiment analysis

As you work with loans and injurences, it might also be useful to use sentiment features in your classification. For example, you could use a dictionary based approach on valence, arousal and dominance and include these features into your classification; you can find information in the relevant paper. Another approach would be to use SentiStrength.

Word2Vec

Recent text analysis approaches have heavily focused on neural network approaches such as Word2Vec that finds word embeddings. Then, a simple approach could be to average the vectors over all words in a document and use these as features. There are also approaches that use the vectors as features in convolutional neural networks.

E-Mail features

Did you think about using further e-mail features, such as the domain, or simply the text length. Might be worth a try to look into that. Also, the domain of e-mail spam classification might give you some further insights.

Feature/classifier tuning

Bag of word/tfidf feature generation is sometimes quite sensitive. You might want to try to tune this by altering the parameters in some form of grid search optimization; for instance, sometimes it is useful to use minimum number of document thresholds for features or things like sublinear scaling. Also, you can try to combine your classifiers in an ensemble approach which might lead to higher accuracy.

Answer 3

Based on what you describe, maybe your problem setting is a multi label setting. E.g: the customer's email might be about an enquiry about two aspects. Say a question about returns policy and price. If that's indeed the case you should try to train multiple binary classifiers like "email about price"and "email not about price" instead of explicitly trading off the categories against each other. If you have $k$ different categories you'll end up with $k$ classifiers. Then while predicting on test data you can take the union of the labels output by the $k$ classifiers as the predicted label.

Please see "PT4" in Multi-Label Classification for more details.