I try to solve a binary text classification problem using sklearn's Tfidf Vecotrizer and a naive bayes classifier. Before I pass the training/test data to the vectorizer I do some text preprocessing. I thought that would increase the performance compared to non preprocessed text. However, the performance did not increase, but actually decreases.
The text that should be classified comes from different PDFs, therefore the text can include unwanted characters/examples like bullet points, dashes if there is a line break or headings. I consider all of that to be noise in my data so I deal with that with some preprocessing.
The text preprocessing consists of the following steps:
- lowercase everything
- remove unwanted characters (punctuation, white spaces, bullet points etc.)
- remove stop words (using spacy's built in stop word list)
Prior to the preprocessing I do a 80/20 train/test split on the data and pass it into a sklearn pipeline consisting of an Tfidf vectorizer and naive bayes classifier. Finally, I run a GridSearch on the pipeline in order to find a good set of params. The performance of the best classifier's performance is reported with classification_report.
In order to see how much of an improvement I gained from the preprocessing, I passed the raw text data to the pipeline. However, the classifier that is trained on the raw data outperforms the classifier that is trained on the processed data.
I assumed that the performance should increase since I kind of normalise my data. Has anyone some advice on why this is the reason? Maybe my assumption is already false or my interpretation of the result is false?