We can use CountVectorizer to count the number of times a word occurs in a corpus:

# Tokenizing text
from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(twenty_train.data)

If we convert this to a data frame we can see what the tokens look like:

For example, the 35,780th word of the 3rd document occurs twice.

We can use TfidfTransformer to count the number of times a word occurs in a corpus (only the term frequency, not the inverse), as follows:

from sklearn.feature_extraction.text import TfidfTransformer
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tf = tf_transformer.transform(X_train_counts)

Converting this to a data frame we get:

We can see the representation is different. The TF is shown as 0.15523. Why is this different than the token count using CountVectorizer?

We can use CountVectorizer to count the number of times a word occurs in a corpus:

# Tokenizing text
from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(twenty_train.data)

If we convert this to a data frame, we can see what the tokens look like:

For example, the 35,780th word of the 3rd document occurs twice.

We can use TfidfTransformer to count the number of times a word occurs in a corpus (only the term frequency and not the inverse) as follows:

from sklearn.feature_extraction.text import TfidfTransformer
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tf = tf_transformer.transform(X_train_counts)

Converting this to a data frame, we get:

We can see the representation is different. The TF is shown as 0.15523. Why is this different than the token count using CountVectorizer?