Text PreProcessing not improving accuracy

Question

I was wondering if TfidfVectorizer() from scikit-learn and its methods fit_transform/transform already do language preprocessing like lowercase/lemmatization/removing punctuation. I am using Imdbs review dump to predict if a review is positive or negative and I wasn't preprocessing initially but recently started to add it in. My accuracy score is 84% using SGDClassifier but whether or not I preprocess the training/testing dataset it seems to be the same?

Answer 1

You can find that information on the class's documentation, here's the link.

Quick summary:

• lowercasing: True by default.
• lemmatization: I think the closest parameter is preprocessor; it's set to None by default but you can specify as the value a callable object of your own and then the vectorizer will apply it on every word to do the lemmatization, e.g.NLTK's WordNet lemmatizer.
• tokenizer: None by default (because the class expects its input to be already tokenized words, so no need for this unless that's not so).
• stopwords: None by default (but you can specify a list of strings as the argument and then the class will filter out any words in that list).

I agree that it is suspicious that you're getting the same Accuracy, generally it is expected to go up when using TFIDF. Make sure you're

1) sending the right input to the TfIdfVectorizer class (it takes a list: list: str, dataset: sentence: each word),

2) using its output (perhaps some variable in your code is still pointing to the non-TFIDF representation instead of the TFIDF one?)

I think it would not be theoretically impossible for SGDClassifier to approximate TFIDF behavior over a sufficiently large number of training iterations with the right learning rate (if all the stopwords' weights are effectively minimized while optimizing loss, TFIDF might have little effect at that point). It would be good to know what parameters you're using and/or to see the code to help you pinpoint the issue.