Text preprocessing decreases classifier accuracy

Question

I try to solve a binary text classification problem using sklearn's Tfidf Vecotrizer and a naive bayes classifier. Before I pass the training/test data to the vectorizer I do some text preprocessing. I thought that would increase the performance compared to non preprocessed text. However, the performance did not increase, but actually decreases.

The text that should be classified comes from different PDFs, therefore the text can include unwanted characters/examples like bullet points, dashes if there is a line break or headings. I consider all of that to be noise in my data so I deal with that with some preprocessing.

The text preprocessing consists of the following steps:

• lowercase everything
• remove unwanted characters (punctuation, white spaces, bullet points etc.)
• remove stop words (using spacy's built in stop word list)

Prior to the preprocessing I do a 80/20 train/test split on the data and pass it into a sklearn pipeline consisting of an Tfidf vectorizer and naive bayes classifier. Finally, I run a GridSearch on the pipeline in order to find a good set of params. The performance of the best classifier's performance is reported with classification_report.

In order to see how much of an improvement I gained from the preprocessing, I passed the raw text data to the pipeline. However, the classifier that is trained on the raw data outperforms the classifier that is trained on the processed data.

I assumed that the performance should increase since I kind of normalise my data. Has anyone some advice on why this is the reason? Maybe my assumption is already false or my interpretation of the result is false?

Answer 1

My guess is that your model overfits, i.e. it learns details which are in the training set by chance. So the model is likely not stable, i.e. it would be very different if you change the training set only a little bit. As a result the performance you obtain is probably meaningless.

• Rule of thumb with text: there should be (much) less features than instances.
• The rare words/features should be discarded ruthlessly, in order to prevent the model from relying on extreme probabilities which happen by chance. This is especially true with Naive Bayes, which tends to overfit very easily.
• You should avoid optimizing min_df, because you can obtain a high performance by chance by taking into account rare words/features (low min_df). It's better to set min_df to a high enough value, e.g. 3 or 4, in order to discard the rare features as explained above.

The target task requires the model to capture the semantics of the full sentence, this might be too hard for a traditional bag-of-words approach. A possibly better approach would be to take into account the syntax, but this is more advanced.