In sklearn tfidf what is the difference between term frequecy and document frequency

Question

Looking at the sklearn tfidf page: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html and trying to understand the difference between term frequency and document frequency. My guess is that term frequency is the number of times the word appears in all the entire corpus, and that document frequency is the number of document or sentences a particular word appears in. For example if I have the following:

corpus = [
    'This is the first line.',
    'This line is the second line.',
    'And this is the third one.',
    'Is this the first line?',
]

The word line appears in three documents or sentences but the total count for the entire corpus is four. So would it be that the document frequency is 3 and the term frequency is 4? Is this correct?

Answer 1

No, regarding the TF-IDF:

• Term frequency (TF): means the count of a term in a specific document
• Document frecuency (DF): means the count of the documents that contains a specific term

At first, I was also confused as I expected a TF-IDF per term, but in reality, you'll have a TF-IDF per term per document. Not all the documents has the same TF-IDF for the same word (i.e. the word line could have different TF-IDF for different documents).

In your example:

• the TF of "line" in doc1 is 1 but in doc2 is 2
• the DF of "line" is 4 (for all corpus)

Based on the following formula ($N$ is number of documents in the corpus): \begin{equation} TF\text{-}IDF = TF\cdot\frac{N}{DF}, \end{equation}

the TF-IDF for the term "line" for these documents are:

• for doc1: \begin{equation} TF\text{-}IDF = 1·\frac{4}{4} = 1, \end{equation}
• for doc2: \begin{equation} TF\text{-}IDF = 2·\frac{4}{4} = 2, \end{equation}

As you can see, the DF is a global metric (is the same for "line" across the corpus), but the TF is specific (could vary for "line" across the corpus).

Answer 2

To add a bit more to @ru.mp 's very good answer.

It is important to understand the terms "document" and "corpus", because these originally referred to Information Retrieval problems (Finding which documents in a corpus contain some terms, like a search engine).

In TF-IDF's use for NLP/Data Mining etc., each of those two terms might have different meanings depending on how your data are structured.

• It may be that "document" is a line from a file and "corpus" is the entire file.
• Or a sentence and a paragraph respectively
• Or an actual document of an actual corpus of course

In your example, the corpus is a single text and each "line" is a document.

No matter which of the above is true, generally in order for the TF-IDF/sklearn combination to make sense, you can imagine your data being in a table, where each row is a document and all the rows together form the corpus. Thus, as @ru.mp said, you can have different values for the same term in different document (as the nominator for a term is different, while the denominator is the same for all term occurences).