2
$\begingroup$

I try to solve a binary text classification problem using sklearn's Tfidf Vecotrizer and a naive bayes classifier. Before I pass the training/test data to the vectorizer I do some text preprocessing. I thought that would increase the performance compared to non preprocessed text. However, the performance did not increase, but actually decreases.

The text that should be classified comes from different PDFs, therefore the text can include unwanted characters/examples like bullet points, dashes if there is a line break or headings. I consider all of that to be noise in my data so I deal with that with some preprocessing.

The text preprocessing consists of the following steps:

  • lowercase everything
  • remove unwanted characters (punctuation, white spaces, bullet points etc.)
  • remove stop words (using spacy's built in stop word list)

Prior to the preprocessing I do a 80/20 train/test split on the data and pass it into a sklearn pipeline consisting of an Tfidf vectorizer and naive bayes classifier. Finally, I run a GridSearch on the pipeline in order to find a good set of params. The performance of the best classifier's performance is reported with classification_report.

In order to see how much of an improvement I gained from the preprocessing, I passed the raw text data to the pipeline. However, the classifier that is trained on the raw data outperforms the classifier that is trained on the processed data.

I assumed that the performance should increase since I kind of normalise my data. Has anyone some advice on why this is the reason? Maybe my assumption is already false or my interpretation of the result is false?

$\endgroup$
4
  • 1
    $\begingroup$ Welcome to DataScienceSE. Questions: how many features and how many instances? What are the parameters optimized by grid search? It's likely overfitting, very common with Naive Bayes. $\endgroup$
    – Erwan
    Sep 11, 2023 at 11:41
  • 1
    $\begingroup$ What are the two classes? In some cases stop words are the best predictors. $\endgroup$
    – gergelybat
    Sep 13, 2023 at 8:01
  • $\begingroup$ @Erwan The document-term matrix is (1182, 5994) for the preprocessed text and (1182, 9182) for the raw text. The parameters I optimise are max_df, min_df, ngram_range and norm for the vectoriser. For the classifier I only optimise the alpha. I was following this tutorial $\endgroup$
    – MC Racoon
    Sep 15, 2023 at 13:32
  • $\begingroup$ @gergelybat I try to classify text that includes some kind of instructions, so e.g. "Place the book on the table" would be class 1 and something like "The sky is bright" would be class 0. I just tried to leave out the stop word removal and now the classifier processed text (i.e. lower cased and removal of special characters) actually outperforms the raw classifier. As you suggested this might hint that too much is removed. I'll try to curate a custom stop word list and see if that gains the desired performance improvement $\endgroup$
    – MC Racoon
    Sep 15, 2023 at 13:51

1 Answer 1

1
$\begingroup$

My guess is that your model overfits, i.e. it learns details which are in the training set by chance. So the model is likely not stable, i.e. it would be very different if you change the training set only a little bit. As a result the performance you obtain is probably meaningless.

  • Rule of thumb with text: there should be (much) less features than instances.
  • The rare words/features should be discarded ruthlessly, in order to prevent the model from relying on extreme probabilities which happen by chance. This is especially true with Naive Bayes, which tends to overfit very easily.
  • You should avoid optimizing min_df, because you can obtain a high performance by chance by taking into account rare words/features (low min_df). It's better to set min_df to a high enough value, e.g. 3 or 4, in order to discard the rare features as explained above.

The target task requires the model to capture the semantics of the full sentence, this might be too hard for a traditional bag-of-words approach. A possibly better approach would be to take into account the syntax, but this is more advanced.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.