Why do we have to remove most common words for text analysis?

Question

I am trying to do sentiment analysis the task is to classify racist tweets from other tweets. And I have read many articles and many have mentioned to remove the most common 10 words from the column because their presence will not of any use in classification of our text data.

So these are my top 10 most common words on my dataset.

[('love', 4271),
 ('day', 3572),
 ('amp', 2709),
 ('happy', 2651),
 ('u', 1840),
 ('time', 1771),
 ('im', 1770),
 ('life', 1756),
 ('like', 1700),
 ('today', 1591)]

If I remove these will my classification model be more accurate?

Similarly they are also recommending to remove the top 10 rare words from the column.

I want to know why? Any help

Answer 1

The simplest way to explain why it may be advantageous to remove the most common words is that they don't give us much information. In your case of classifying racist tweets, words like "and", "a", "the", etc. don't help the classifier and may act as noise which negatively impacts performance.

I would not say that the removal of the n most popular words will guarantee that the model will be more accurate, but it is a parameter you can explore. Outside of complete removal of the most common words, you may want to look into techniques like subsampling.

Answer 2

Stop words wont give you any insights and further there are frequently used in any text so that frequency of such words are higher than other useful words in your text. This will results into giving more weight age to the stop words then other words. This will affect the performance of the model especially when you apply algorithms based of TF-IDF( Term Frequency- Inverse Document Frequency).

For example: Consider below paragraph of text on which I want to apply Text Analytics algorithm.

** Hi my name is X.My home town is Y. My favorite dish is Pasta. I like Sachin . He is the greatest cricketer of all time. **

Here, frequency of words like is - 3 my - 3 and so on.

So while extracting most important text to summarize large chunk of data, algorithm consider is as an important word because it has been repeated many times. Obviously to summarize the text what I would be looking at words like Sachin , town , favorite and not words like is and my.

Answer 3

Great question Sam.

As others have mentioned, stop words such as "a", "having", and "they" cause a litany of issues when it comes to text analysis:

• They don't help identify what is going in in a document. If I told you the word "me" appears 12 times in a document with 500 words, you wouldn't be able to confidently make any statement about what that document is about, or if it's similar to another document with the word "me" in it.

• They really hurt computational speed. These words show up very often, and if we don't remove them our algorithm is going to use them to analyze the documents. If you could reduce the size of your data by 25%, just by removing stop words, it could drastically improve performance.

In addition to common stop words, it may be important to define your own stop words. In your case, I can't readily think of any new stop word's you'd want to define (maybe just the word "tweet" or "twitter"). However, if you were looking at tweets about wine, and 99% of the tweets had the word "wine" in it, then that's a word you'd definitely want to remove.

Hope this helps!