0
$\begingroup$

I am trying to understand what happens inside the IDF part of the TFIDF vectorizer.

SKlearn tfidf vectorizer

The official sci-kit Learn page says that the shape is (4,9) for a corpus of 4 documents having 9 unique features.

I get the Term Frequency (TF) part, it makes sense to me that ( for every unique feature(9), for each document(4) we calculate each term's frequency, so we get a matrix of shape (4,9)

But what doesn't make sense to me is the IDF part the formula for IDF is

idf formula

So applying this formula, for every feature (9) we calculate the log ( total no.of documents / no.of docs having the term or feature in it) I think This will result in a shape of (1,9) , please correct my understanding here.

Improve this question
$\endgroup$
0
$\begingroup$

The inverse document frequency transformation of TFIDF does not affect the shape of your vector. The shape is only altered during the tokenization phase which, in your case, results in a shape of (4, 9). The inverse document frequency portion only scales your existing features in accordance with how frequently these tokens arise elsewhere in your corpus. This can be thought as a one-to-one mapping of your original term-frequency vector (the tokenized corpus) to the tfidf vector (scaled using information on how frequent your vectors tokens exist elsewhere in your corpus).

Improve this answer
$\endgroup$
4
  • $\begingroup$ I don't seem to understand it completely as i am new to Machine Learning.I want to implement my version of the tf-idf vectorizer in python. Here we have a corpus of 4 documents, after i do the verctorizer.fit() , i get 9 unique words/features. now i am trying to find the term-frequency and inverse-document-frequency so i can multiply both matrix to create the tf - idf vectorizer , the result of my term-frequency is a (4,9) matrix , i am stuck when trying to find the matrix of idf, my guess is, it will be a (1,9) matrix , how can i multiply both ? $\endgroup$ – Allan_Aj5 Nov 2 '20 at 7:03
  • $\begingroup$ (1,9) because for every unique feature i will find the log(total no.of documents / no.of docs that have the feature) $\endgroup$ – Allan_Aj5 Nov 2 '20 at 7:03
  • $\begingroup$ Take a look at this for a walkthrough: towardsdatascience.com/… $\endgroup$ – Oliver Foster Nov 2 '20 at 15:56
  • $\begingroup$ Thanks a lot for the quick response and reference material $\endgroup$ – Allan_Aj5 Nov 5 '20 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.