Are stopwords helpful when using tf-idf features for document classification?

Question

I have documents of pure natural language text. Those documents are rather short; e.g. 20 - 200 words. I want to classify them.

A typical representation is a bag of words (BoW). The drawback of BoW features is that some features might always be present / have a high value, simply because they are an important part of the language. Stopwords like the following are examples: is, are, with, the, a, an, ...

One way to deal with that is to simply define this list and remove them, e.g. by looking at the most common words and just deciding which of them don't carry meaning for the given task. Basically by gut feeling.

Another way is TF-IDF features. They weight the words by how often they occur in the training set overall vs. how often they occur in the specific document. This way, even words which might not directly carry meaningful information might be valuable.

The last part is my question: Should I remove stopwords when I use TF-IDF features? Are there any publications on this topic? (I'm pretty sure I'm not the first one to wonder about this question)

Answer 1

From the way the TfIdf score is set up, there shouldn't be any significant difference in removing the stopwords. The whole point of the Idf is exactly to remove words with no semantic value from the corpus. If you do add the stopwords, the Idf should get rid of it.

However, working without the stopwords in your documents will make the number of features smaller, which might have a slight computational advantage.

Answer 2

Term frequency-inverse document frequency (TF-IDF) can be sensitive to the presence of stop words, which are common words that occur frequently in most documents and do not provide useful information. Because TF-IDF is based on the frequency of words in a document, it can give higher weights to stop words if they occur frequently in the document. This can result in stop words having a disproportionate influence on the overall representation of the document, which can be detrimental to the performance of the model.

To mitigate this issue, it is common to remove stop words from the documents before calculating the TF-IDF vectors. This can help to reduce the influence of stop words on the vectors and improve the performance of the model. In some cases, it may also be useful to apply other techniques to reduce the dimensionality of the vectors, such as singular value decomposition (SVD) or principal component analysis (PCA), to remove less important features and improve the quality of the vectors.

Answer 3

In addition to Icaro's answer, please refer to this link.

Pros/Cons of stop word removal?

Answer 4

I'd agree that the tf-idf score won't be overly susceptible to stop words.

But still removing them can be beneficial. In particular with low numbers of documents to learn on. As it reduces the dimensionality of your input space.

Can always try both.

Dependent on your method for classification you can get feature importances (like naive bayes) and then remove words that aren't significant.