2
$\begingroup$

I have a dataframe:

df = pd.DataFrame({'docs': ['gamma alfa beta beta epsilon', 'beta gamma eta',], 'labels': ['alfa alfa beta', 'gamma fi']})

I use count vectorizer:

import numpy as np
import pandas as pd
from itertools import chain
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

vocab_docs = set(chain(*[i.split() for i in df['docs'].unique()]))
cv_docs = CountVectorizer(vocabulary=vocab_docs)
cv_docs_s = cv_docs.fit_transform(df['docs'])

I do TFIDF:

tfidf_docs = TfidfVectorizer(vocabulary=vocab_docs)
tfidf_docs_s = tfidf_docs.fit_transform(df['docs'])
# tfidf docs
tfidf_docs_s = tfidf_docs_s.todense()

but I see that the results are different:

test = np.multiply(cv_docs_s.todense(), tfidf_docs.idf_)

test != tfidf_docs_s 

Why is the result of CountVectorizer * TfidfVectorizer.idf_ different from TfidfVectorizer.fit_transform()?

$\endgroup$
1
$\begingroup$

TfidfVectorizer will by default normalize each row. From the documentation we can see that:

norm : ‘l1’, ‘l2’ or None, optional (default=’l2’)
Each output row will have unit norm, either: * ‘l2’: Sum of squares of vector elements is 1. The cosine similarity between two vectors is their dot product when l2 norm has been applied. * ‘l1’: Sum of absolute values of vector elements is 1. See preprocessing.normalize

Setting norm to None will give the result you expect:

tfidf_docs = TfidfVectorizer(vocabulary=vocab_docs, norm=None)
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.