I am creating the information gains of all the words present in the vocabulary. However, when I check for feature names of the vectorizer it is different.
For bag of words I am using:
total_words = df['description'].str.split(expand = True).stack().value_counts() total_words = total_words.index.tolist()
For vectorizer I am using:
vect = TfidfVectorizer() tfidf = vect.fit_transform(df['description']) vect.get_feature_names()
I am removing words that have length < 3. However, I am seeing 2, 3 letter word in the
feature_names, but when I check this in the
total_words everything is fine.