# In sklearn tfidf what is the difference between term frequecy and document frequency

Looking at the sklearn tfidf page: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html and trying to understand the difference between term frequency and document frequency. My guess is that term frequency is the number of times the word appears in all the entire corpus, and that document frequency is the number of document or sentences a particular word appears in. For example if I have the following:

corpus = [
'This is the first line.',
'This line is the second line.',
'And this is the third one.',
'Is this the first line?',
]


The word line appears in three documents or sentences but the total count for the entire corpus is four. So would it be that the document frequency is 3 and the term frequency is 4? Is this correct?

No, regarding the TF-IDF:

• Term frequency (TF): means the count of a term in a specific document
• Document frecuency (DF): means the count of the documents that contains a specific term

At first, I was also confused as I expected a TF-IDF per term, but in reality, you'll have a TF-IDF per term per document. Not all the documents has the same TF-IDF for the same word (i.e. the word line could have different TF-IDF for different documents).

• the TF of "line" in doc1 is 1 but in doc2 is 2
• the DF of "line" is 4 (for all corpus)

Based on the following formula ($$N$$ is number of documents in the corpus): $$\begin{equation} TF\text{-}IDF = TF\cdot\frac{N}{DF}, \end{equation}$$

the TF-IDF for the term "line" for these documents are:

• for doc1: $$\begin{equation} TF\text{-}IDF = 1·\frac{4}{4} = 1, \end{equation}$$
• for doc2: $$\begin{equation} TF\text{-}IDF = 2·\frac{4}{4} = 2, \end{equation}$$

As you can see, the DF is a global metric (is the same for "line" across the corpus), but the TF is specific (could vary for "line" across the corpus).

• Thank you very much for your answer! I'm still a little confused because the page: scikit-learn.org/stable/modules/generated/… says that the top max_features are "ordered by term frequency". Shouldn't they be ordered by document frequency? Dec 7, 2022 at 20:58
• I think you're mixing up TF-IDF vectos with Vocabulary. The first is what you receive when transforming the input, the latter is a dict that contains the vocabulary of your corpus. Sci-kit Learn says: If not None, build a vocabulary that only consider the top max_features ordered by term frequency across the corpus. The vocabulary is a dict with terms as the keys and frequency of the terms (across the entire corpus) as the values. Then the vocabulary is ordered by values and filtered out by max_features. Finally, build the TF-IDF vectors with just the terms in the Vocabulary. Dec 8, 2022 at 15:50