I was wondering if TfidfVectorizer() from scikit-learn and its methods fit_transform/transform already do language preprocessing like lowercase/lemmatization/removing punctuation. I am using Imdbs review dump to predict if a review is positive or negative and I wasn't preprocessing initially but recently started to add it in. My accuracy score is 84% using SGDClassifier but whether or not I preprocess the training/testing dataset it seems to be the same?


1 Answer 1


You can find that information on the class's documentation, here's the link.

Quick summary:

  • lowercasing: True by default.
  • lemmatization: I think the closest parameter is preprocessor; it's set to None by default but you can specify as the value a callable object of your own and then the vectorizer will apply it on every word to do the lemmatization, e.g.NLTK's WordNet lemmatizer.
  • tokenizer: None by default (because the class expects its input to be already tokenized words, so no need for this unless that's not so).
  • stopwords: None by default (but you can specify a list of strings as the argument and then the class will filter out any words in that list).

I agree that it is suspicious that you're getting the same Accuracy, generally it is expected to go up when using TFIDF. Make sure you're

1) sending the right input to the TfIdfVectorizer class (it takes a list: list: str, dataset: sentence: each word),

2) using its output (perhaps some variable in your code is still pointing to the non-TFIDF representation instead of the TFIDF one?)

I think it would not be theoretically impossible for SGDClassifier to approximate TFIDF behavior over a sufficiently large number of training iterations with the right learning rate (if all the stopwords' weights are effectively minimized while optimizing loss, TFIDF might have little effect at that point). It would be good to know what parameters you're using and/or to see the code to help you pinpoint the issue.

  • 1
    $\begingroup$ github.com/Tibblist/nlp-imdb/blob/master/imdb_sentiment.py This is the code I am using and how I am setting it up, I'm just passing in paragraphs of text as a single string so maybe I am incorrectly setting up the tfidfvector. $\endgroup$ Commented Jul 4, 2019 at 1:42
  • $\begingroup$ If you're passing in paragraphs (so, a list of strings) to TfIdfVectorizer's fit or fit_transform methods, then you're totally fine -the class then uses the specified tokenizer to generate the list: list: str representation used internally I mentioned in my answer, so that should be happening in your case, which means the class is receiving the right input. I couldn't see the code, though (I think the repository is private), so I still don't have a good answer for your question. $\endgroup$ Commented Jul 4, 2019 at 7:06
  • $\begingroup$ Made it public but that definitely makes sense, I just didn't know where and how scikit-learn would be changing the text I passed in. $\endgroup$ Commented Jul 4, 2019 at 16:01
  • $\begingroup$ Thank you! The code looks good, I couldn't find any obvious issues. I ran it with both TfidfVectorizer and CountVectorizer and I did get different results: 84% and 87% accuracy, respectively. So, the behavior I am getting seems consistent and I couldn't reproduce the exact issue you described. Does your other implementation use a CountVectorizer? Or just a different classifier? I think it should work for you too (I assume you're using the ai.stanford.edu/~amaas/data/sentiment corpus). $\endgroup$ Commented Jul 5, 2019 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.