2
$\begingroup$

What are the pros / cons of removing stop words from text in the context of a text classification problem, I'm wondering what the best approach is (i.e. to remove or not to remove)?

I've read somewhere (but can't locate the reference) that it may be detrimental the the performance of a model in the case of sentiment analysis to remove stop words.

$\endgroup$
  • $\begingroup$ Pro is that it helps the model to get the root words which are important rather than focusing on quite famous and commonly used words... $\endgroup$ – Aditya Apr 30 '18 at 17:40
1
$\begingroup$

In the context of sentiment analysis, removing stop words can be problematic if context is affected. For example suppose your stop word corpus includes ‘not’, which is a negation that can alter the valence of the passage. So you have to be cautious of exactly what is being dropped and what consequences it can have.

$\endgroup$
1
$\begingroup$

If you are using some bag of words based methods, i.e, countVectorizer or tfidf that works on counts and frequency of the words, removing stopwords is great as it lowers the dimensional space and also a few stop words won't drive your analysis. On the other hand, when you are exploiting the semantics of the given text, say in a seq2seq model, removing stopwords will omit the context and you will end up with ambiguous results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.