3
$\begingroup$

I have documents of pure natural language text. Those documents are rather short; e.g. 20 - 200 words. I want to classify them.

A typical representation is a bag of words (BoW). The drawback of BoW features is that some features might always be present / have a high value, simply because they are an important part of the language. Stopwords like the following are examples: is, are, with, the, a, an, ...

One way to deal with that is to simply define this list and remove them, e.g. by looking at the most common words and just deciding which of them don't carry meaning for the given task. Basically by gut feeling.

Another way is TF-IDF features. They weight the words by how often they occur in the training set overall vs. how often they occur in the specific document. This way, even words which might not directly carry meaningful information might be valuable.

The last part is my question: Should I remove stopwords when I use TF-IDF features? Are there any publications on this topic? (I'm pretty sure I'm not the first one to wonder about this question)

$\endgroup$
3
$\begingroup$

From the way the TfIdf score is set up, there shouldn't be any significant difference in removing the stopwords. The whole point of the Idf is exactly to remove words with no semantic value from the corpus. If you do add the stopwords, the Idf should get rid of it.

However, working without the stopwords in your documents will make the number of features smaller, which might have a slight computational advantage.

| improve this answer | |
$\endgroup$
0
$\begingroup$

In addition to Icaro's answer, please refer to this link.

Pros/Cons of stop word removal?

| improve this answer | |
$\endgroup$
0
$\begingroup$

I'd agree that the tf-idf score won't be overly susceptible to stop words.

But still removing them can be beneficial. In particular with low numbers of documents to learn on. As it reduces the dimensionality of your input space.

Can always try both.

Dependent on your method for classification you can get feature importances (like naive bayes) and then remove words that aren't significant.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.