2
$\begingroup$

Looking at the sklearn tfidf page: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html and trying to understand the difference between term frequency and document frequency. My guess is that term frequency is the number of times the word appears in all the entire corpus, and that document frequency is the number of document or sentences a particular word appears in. For example if I have the following:

corpus = [
    'This is the first line.',
    'This line is the second line.',
    'And this is the third one.',
    'Is this the first line?',
]

The word line appears in three documents or sentences but the total count for the entire corpus is four. So would it be that the document frequency is 3 and the term frequency is 4? Is this correct?

$\endgroup$

2 Answers 2

3
$\begingroup$

No, regarding the TF-IDF:

  • Term frequency (TF): means the count of a term in a specific document
  • Document frecuency (DF): means the count of the documents that contains a specific term

At first, I was also confused as I expected a TF-IDF per term, but in reality, you'll have a TF-IDF per term per document. Not all the documents has the same TF-IDF for the same word (i.e. the word line could have different TF-IDF for different documents).

In your example:

  • the TF of "line" in doc1 is 1 but in doc2 is 2
  • the DF of "line" is 4 (for all corpus)

Based on the following formula ($N$ is number of documents in the corpus): \begin{equation} TF\text{-}IDF = TF\cdot\frac{N}{DF}, \end{equation}

the TF-IDF for the term "line" for these documents are:

  • for doc1: \begin{equation} TF\text{-}IDF = 1·\frac{4}{4} = 1, \end{equation}
  • for doc2: \begin{equation} TF\text{-}IDF = 2·\frac{4}{4} = 2, \end{equation}

As you can see, the DF is a global metric (is the same for "line" across the corpus), but the TF is specific (could vary for "line" across the corpus).

$\endgroup$
2
  • $\begingroup$ Thank you very much for your answer! I'm still a little confused because the page: scikit-learn.org/stable/modules/generated/… says that the top max_features are "ordered by term frequency". Shouldn't they be ordered by document frequency? $\endgroup$
    – james pow
    Commented Dec 7, 2022 at 20:58
  • $\begingroup$ I think you're mixing up TF-IDF vectos with Vocabulary. The first is what you receive when transforming the input, the latter is a dict that contains the vocabulary of your corpus. Sci-kit Learn says: If not None, build a vocabulary that only consider the top max_features ordered by term frequency across the corpus. The vocabulary is a dict with terms as the keys and frequency of the terms (across the entire corpus) as the values. Then the vocabulary is ordered by values and filtered out by max_features. Finally, build the TF-IDF vectors with just the terms in the Vocabulary. $\endgroup$
    – ru.mp
    Commented Dec 8, 2022 at 15:50
1
$\begingroup$

To add a bit more to @ru.mp 's very good answer.

It is important to understand the terms "document" and "corpus", because these originally referred to Information Retrieval problems (Finding which documents in a corpus contain some terms, like a search engine).

In TF-IDF's use for NLP/Data Mining etc., each of those two terms might have different meanings depending on how your data are structured.

  • It may be that "document" is a line from a file and "corpus" is the entire file.
  • Or a sentence and a paragraph respectively
  • Or an actual document of an actual corpus of course

In your example, the corpus is a single text and each "line" is a document.

No matter which of the above is true, generally in order for the TF-IDF/sklearn combination to make sense, you can imagine your data being in a table, where each row is a document and all the rows together form the corpus. Thus, as @ru.mp said, you can have different values for the same term in different document (as the nominator for a term is different, while the denominator is the same for all term occurences).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.