0
$\begingroup$

I am creating the information gains of all the words present in the vocabulary. However, when I check for feature names of the vectorizer it is different.

For bag of words I am using:

total_words = df['description'].str.split(expand = True).stack().value_counts()
total_words = total_words.index.tolist()

For vectorizer I am using:

vect = TfidfVectorizer()
tfidf = vect.fit_transform(df['description'])
vect.get_feature_names()

I am removing words that have length < 3. However, I am seeing 2, 3 letter word in the feature_names, but when I check this in the total_words everything is fine.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.