I am creating the information gains of all the words present in the vocabulary. However, when I check for feature names of the vectorizer it is different.
For bag of words I am using:
total_words = df['description'].str.split(expand = True).stack().value_counts()
total_words = total_words.index.tolist()
For vectorizer I am using:
vect = TfidfVectorizer()
tfidf = vect.fit_transform(df['description'])
vect.get_feature_names()
I am removing words that have length < 3. However, I am seeing 2, 3 letter word in the feature_names, but when I check this in the total_words everything is fine.
