Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
What are the pros / cons of removing stop words from text in the context of a text classification problem, I'm wondering what the best approach is (i.e. to remove or not to remove)?
I've read somewhere (but can't locate the reference) that it may be detrimental the the performance of a model in the case of sentiment analysis to remove stop words.
In the context of sentiment analysis, removing stop words can be problematic if context is affected. For example suppose your stop word corpus includes ‘not’, which is a negation that can alter the valence of the passage. So you have to be cautious of exactly what is being dropped and what consequences it can have.
If you are using some bag of words based methods, i.e, countVectorizer or tfidf that works on counts and frequency of the words, removing stopwords is great as it lowers the dimensional space and also a few stop words won't drive your analysis. On the other hand, when you are exploiting the semantics of the given text, say in a seq2seq model, removing stopwords will omit the context and you will end up with ambiguous results.
