We can use CountVectorizer to count the number of times a word occurs in a corpus: # Tokenizing text from sklearn.feature_extraction.text import CountVectorizer count_vect = CountVectorizer() X_train_counts = count_vect.fit_transform(twenty_train.data) If we convert this to a data frame, we can see what the tokens look like: [![enter image description here][1]][1] For example, the 35,780th word of the 3rd document occurs twice. We can use TfidfTransformer to count the number of times a word occurs in a corpus (only the term frequency and not the inverse) as follows: from sklearn.feature_extraction.text import TfidfTransformer tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts) X_train_tf = tf_transformer.transform(X_train_counts) Converting this to a data frame, we get: [![enter image description here][2]][2] [1]: https://i.sstatic.net/eB4Uh.png [2]: https://i.sstatic.net/3VnIe.png We can see the representation is different. The TF is shown as 0.15523. Why is this different than the token count using CountVectorizer?