Why is the result of CountVectorizer * TfidfVectorizer.idf_ different from TfidfVectorizer.fit_transform()?

Question

I have a dataframe:

df = pd.DataFrame({'docs': ['gamma alfa beta beta epsilon', 'beta gamma eta',], 'labels': ['alfa alfa beta', 'gamma fi']})

I use count vectorizer:

import numpy as np
import pandas as pd
from itertools import chain
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

vocab_docs = set(chain(*[i.split() for i in df['docs'].unique()]))
cv_docs = CountVectorizer(vocabulary=vocab_docs)
cv_docs_s = cv_docs.fit_transform(df['docs'])

I do TFIDF:

tfidf_docs = TfidfVectorizer(vocabulary=vocab_docs)
tfidf_docs_s = tfidf_docs.fit_transform(df['docs'])
# tfidf docs
tfidf_docs_s = tfidf_docs_s.todense()

but I see that the results are different:

test = np.multiply(cv_docs_s.todense(), tfidf_docs.idf_)

test != tfidf_docs_s 

Why is the result of CountVectorizer * TfidfVectorizer.idf_ different from TfidfVectorizer.fit_transform()?

Answer 1

TfidfVectorizer will by default normalize each row. From the documentation we can see that:

norm : ‘l1’, ‘l2’ or None, optional (default=’l2’)
Each output row will have unit norm, either: * ‘l2’: Sum of squares of vector elements is 1. The cosine similarity between two vectors is their dot product when l2 norm has been applied. * ‘l1’: Sum of absolute values of vector elements is 1. See preprocessing.normalize

Setting norm to None will give the result you expect:

tfidf_docs = TfidfVectorizer(vocabulary=vocab_docs, norm=None)