I am creating the information gains of all the words present in the vocabulary. However, when I check for feature names of the vectorizer it is different.

For bag of words I am using:

total_words = df['description'].str.split(expand = True).stack().value_counts()
total_words = total_words.index.tolist()

For vectorizer I am using:

vect = TfidfVectorizer()
tfidf = vect.fit_transform(df['description'])

I am removing words that have length < 3. However, I am seeing 2, 3 letter word in the feature_names, but when I check this in the total_words everything is fine.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.