0
$\begingroup$

I am trying to understand what happens inside the IDF part of the TFIDF vectorizer.

SKlearn tfidf vectorizer

The official scikit-learn page says that the shape is (4,9) for a corpus of 4 documents having 9 unique features.

I get the Term Frequency (TF) part, it makes sense to me that ( for every unique feature(9), for each document(4) we calculate each term's frequency, so we get a matrix of shape (4,9)

But what does not make sense to me is the IDF part the formula for IDF is:

$$\text{idf}(t,D) = \text{log} \ {{N} \over {| \{ d \in D:t \in d \} | }}$$

with

  • $N$: total number of documents in the corpus $N = |D|$

  • $| \{ d \in D:t \in d \} |$ : number of documents where the term $t$ appears (i.e., $\text{tf}(t,d) \neq 0$). If the term is not in the corpus, this will lead to a division-by-zero. It is therefore common to adjust the denominator to $1 + | \{ d \in D:t \in d \} |$.

So applying this formula, for every feature (9) we calculate the log(total number of documents / number of documents having the term or feature in it) I think This will result in a shape of (1,9), please correct my understanding here.

$\endgroup$

1 Answer 1

0
$\begingroup$

The inverse document frequency transformation of TFIDF does not affect the shape of your vector. The shape is only altered during the tokenization phase which, in your case, results in a shape of (4, 9). The inverse document frequency portion only scales your existing features in accordance with how frequently these tokens arise elsewhere in your corpus. This can be thought as a one-to-one mapping of your original term-frequency vector (the tokenized corpus) to the tfidf vector (scaled using information on how frequent your vectors tokens exist elsewhere in your corpus).

$\endgroup$
4
  • $\begingroup$ I don't seem to understand it completely as i am new to Machine Learning.I want to implement my version of the tf-idf vectorizer in python. Here we have a corpus of 4 documents, after i do the verctorizer.fit() , i get 9 unique words/features. now i am trying to find the term-frequency and inverse-document-frequency so i can multiply both matrix to create the tf - idf vectorizer , the result of my term-frequency is a (4,9) matrix , i am stuck when trying to find the matrix of idf, my guess is, it will be a (1,9) matrix , how can i multiply both ? $\endgroup$
    – Allan_Aj5
    Nov 2, 2020 at 7:03
  • $\begingroup$ (1,9) because for every unique feature i will find the log(total no.of documents / no.of docs that have the feature) $\endgroup$
    – Allan_Aj5
    Nov 2, 2020 at 7:03
  • $\begingroup$ Take a look at this for a walkthrough: towardsdatascience.com/… $\endgroup$ Nov 2, 2020 at 15:56
  • $\begingroup$ Thanks a lot for the quick response and reference material $\endgroup$
    – Allan_Aj5
    Nov 5, 2020 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.