While trying to remove stopwords using the nltk package, the following error occurred:

from tqdm import tqdm 
import nltk
from nltk.corpus import stopwords

preprocessed_reviews = []
for sentance in tqdm(final["Text"].values):
    sentance = ' '.join(e.lower() for e in sentance.split() if e.lower() not in stopwords)
  Traceback (most recent call last)
<ipython-input-136-ac5c19fafd9c> in <module>()
---> 7     sentance = ' '.join(e.lower() for e in sentance.split() if e.lower() not in stopwords)
     8     preprocessed_reviews.append(sentance.strip())

TypeError: argument of type 'LazyCorpusLoader' is not iterable

There are a couple of items that could be improved in your code:

  • nltk.corpus.stopwords is a nltk.corpus.util.LazyCorpusLoader. You might want stopwords.words('english'), a list of stop English words.
  • It can cause bugs to update a variable as you iterate through it, for example sentance in your code.
  • In your code preprocessed_reviews is not being updated.
  • You might want to tokenize instead of str.split().

Here is a revision version:

import nltk
from nltk.corpus import stopwords
from tqdm import tqdm

reviews_raw = ('The quick brown fox jumps over the lazy dog', "I have been doing what I should not been doing")
reviews_processed = []
for sentence in tqdm(reviews_raw):
    reviews_processed.append(' '.join(token.lower() for token in nltk.word_tokenize(sentence) if token.lower() not in stopwords.words('english')))

assert reviews_processed == ['quick brown fox jumps lazy dog', '']
  • $\begingroup$ Yes, The above improvements helped execute the code without issue. split() didnot cause any issue but tokenizing is a good concept here in this case. Thank you. ! $\endgroup$ – hari charan vihari Jun 8 '19 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.