1
$\begingroup$

I am trying to understand why tf-idf is useful. As I understand the formula to work out the tf-idf is:

enter image description here

Can someone explain what is wrong with the reasoning below:

Imagine I have 100 documents that have a text corpus of 10,000 words. As I understand each document will have 10,000 features once transformed using tf-idf. For a single feature we get the tf part by counting the number of occurrences of that word in each document. We then multiply that feature by the idf.

TL;DR My question is given that we multiply the entire feature by a constant (the value of log(N/dfi)), how does that help in making a model better. As I understand multiplying a feature by a constant doesn't help as the model can figure out this new scale.

Edit After applying tf we might get a document-term matrix like the following:

word / 
doc num. computer walk smell help warmth
1        1        0    2     1    0     
2        0        2    3     1    1
3        0        1    0     0    3
4        1        2    0     1    0
5        0        1    2     2    1

Then after tf-idf we might get the following:

word / 
doc num. computer    walk        smell      help        warmth
1        1*log(5/2)  0*log(5/4)  2*log(5/3) 1*log(5/4)  0*log(5/3)  
2        0*log(5/2)  2*log(5/4)  3*log(5/3) 1*log(5/4)  1*log(5/3)
3        0*log(5/2)  1*log(5/4)  0*log(5/3) 0*log(5/4)  3*log(5/3)
4        1*log(5/2)  2*log(5/4)  0*log(5/3) 1*log(5/4)  0*log(5/3)
5        0*log(5/2)  1*log(5/4)  2*log(5/3) 2*log(5/4)  1*log(5/3)

tf-idf seems to just scale each feature here by the same value so why does that help certain models like decision trees that can handle different scaling to predict better?

Improve this question
$\endgroup$
7
  • $\begingroup$ log(N/dfi)) is not constant at all, It will depend on the specific term, the only constant will be N (the number of documents) So you weight each term according to its pretense across documents. A term that appears in all documents will be consider lees important (log(1) = 0) This is specially evident when you have stop words in your documents (you have not cleaned your corpus) since stop words will appear in almost all documents $\endgroup$
    – Multivac
    Feb 18, 2021 at 16:55
  • $\begingroup$ But it doesn't depend on the document. Every document will get multiplied by the same value for that feature $\endgroup$
    – Daniel Wyatt
    Feb 18, 2021 at 16:57
  • $\begingroup$ But remember that your features on this vectorization will be the words not the documents, you want to give weight to each word(feature) the documents are your observations. I do not know if I'm being clear $\endgroup$
    – Multivac
    Feb 18, 2021 at 17:00
  • $\begingroup$ TL;DR My question is given that we multiply the entire feature by a constant You do not, because your features are the words not the documents $\endgroup$
    – Multivac
    Feb 18, 2021 at 17:02
  • $\begingroup$ Yes I agree with that. But in a normal model (say linear regression) I wouldn't think that changing a feature by multiplying it by a constant is that useful. Why isn't that the case here? $\endgroup$
    – Daniel Wyatt
    Feb 18, 2021 at 17:03

2 Answers 2

Reset to default
0
$\begingroup$

In a classical ML approach i.e. making each word as a feature, it will not make any difference if we ignore the burden of having extra uninformative features.
At the end, each feature is simply standardized by a constant (separate for each feature).

It can help in -

  • Feature engineering - If we have to pick the top 1000 features, it can be based on tf-idf instead of count. e.g. removing stop words

  • Searching documents on query e.g. ignoring "The" in "The Brown Cow" as explained here Wikipedia

Improve this answer
$\endgroup$
1
$\begingroup$

To complement my comment I'm taking those paragraphs from data camp tutorial in which they explain this in a very clear way

enter image description here

Improve this answer
$\endgroup$
1
  • $\begingroup$ I don't think this is true. When it comes to modelling using decision trees, longer documents will still carry more weight than shorter ones. $\endgroup$
    – Daniel Wyatt
    Feb 22, 2021 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.