Text Classifier with multiple bag-of-words

Question

I am training an email classifier from a dataset with separate columns for both the subject line and the content of the email itself. I've pre-processed the content column in such a way that the subject and associated metadata have been completely removed. My thinking, at this point, is that I should train the classifier on two separate bag-of-words, one for the subject column and one for the content column. Is this the right approach? Is it there a simple way to accomplish this using NLTK or a similar library?

Answer 1

I cannot comprehend the question completely but let me see if I got this right.

You want to train a classifier. You have different columns for subject and body. You have now pre-processed it in a manner such that subject and other meta information have been removed.

Now I shall try to answer your question. Keep in mind that, this is how I would approach the problem (based on experience with similar task in the past) but there could be better ways or approaches.

I would not remove the subject column in the first place. I would also not train the classifier on two separate bag-of-words. I would append both, the subject and the content, into one single document and then classify them accordingly with one unit of bag-of-words instead of two.

Why would I do this? There could be times when the subject could have keywords or something that could lead to better classification than the body of the email. The vice versa could also be true. Hence, without having seen the entire dataset, and to be on the safer side, I would also give this approach a shot. Compare both of your approaches and then see where you have better scores of accuracy, precision, recall and other metrics that aids you in your conclusions.

Ways to accomplish this? Yes, NLTK is one way to get the bag of words. However, I would go ahead with scikit-learn package to train my classifier. I would also run LSA/LDA/HDP (Topic Modeling Algorithms) to see what constitutes my data. To do this, I would use Gensim.

http://scikit-learn.org/stable/

https://radimrehurek.com/gensim/

Additional tutorials to help you get started with text classification if you are new to it (that is what it seems like, pardon me if I am wrong). These will also help you in achieving better results to begin with:

https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a

http://zacstewart.com/2015/04/28/document-classification-with-scikit-learn.html

Although both the tutorials I mentioned primarily concentrate on binary classification, you can also achieve fine-grained (multi-class) classification in similar manner.

Answer 2

As I cannot comment @trollster's good response: for appending two bag-of-words or custom features extracted from the text you can use FeatureUnion in scikit-learn. They provide a good example in their doc : http://scikit-learn.org/stable/auto_examples/hetero_feature_union.html#sphx-glr-auto-examples-hetero-feature-union-py

In fact, their example treat a quite similar problem to yours (20 newsgroup dataset which consists in blog posts). As @trollster said, I would not separate subject and content as they might be highly related and contribute equally to the classification.

Answer 3

Your intuition that the distributions of words in the subject and email body are likely to be different has merit. Regarding and "out of the box" solution, the easiest approach would be to construct two separate classifiers -- one each for the subject and body -- and then ensemble them together. A very simple ensemblification approach would be to just use the probability scores outputted by those two models as inputs to a logistic regression classifier.

Answer 4

Two different bags of words would not make sense here, but If you can use two different word embeddings and then use these with Functional APIs using a Neural network, it should improve the performance of the model.