3
$\begingroup$

Let's suppose that I have a dataset of 1,000 documents.

Each document is a restaurant review (so relatively short text) and it has labels {Negative, Indifferent, Positive}.

Let's suppose that the dataset has 600 positive reviews, 200 indifferent reviews and 200 negative reviews.

I want to train a classifier to classify a review as Negative or Indifferent or Positive based on the text of the review.

Without using any word embeddings for now the best way to go is to use a TF-IDF as feature engineering.

However, as I am thinking this more carefully I am not entirely surely if this is the best way to go in comparison with a simply term-frequency (tf) model.

Specifically, a TF-IDF model will take the inverse document frequency of terms irrespectively to any labels/categories.

So in the example above if many of the positive reviews have the word 'positive' in them then this word will automatically have modified (and in general lower) TF-IDF scores simply because the majority of the documents in the dataset are positive (600 documents).

On the other hand, with a simply TF then the word 'positive' would have a very high value and it would be evident that it is directly related to positive reviews.

Why the TF-IDF is necessarily the best way to go in cases like these?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

So in the example above if many of the positive reviews have the word 'positive' in them then this word will automatically have modified (and in general lower) TF-IDF scores simply because the majority of the documents in the dataset are positive (600 documents).

You're thinking of the word positive as example because you are a human and you know what to select. The words which have the highest frequency in English are stop words: the, a, is... These are useless and would add a lot of noise in your model. You can remove them using a predefined list of stop words, but what about words which are not stop words but are frequent enough, for instance food, place, chef... This is where IDF helps.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Maybe, maybe but ultimately it will also affect words such as 'positive' etc. To be honest, finally it is perhaps only a matter of testing; test both models and see what happens but I was just wondering because TF-IDF is almost always considered better than TF (and I am not absolutely sure about it - there is no proof in any case). $\endgroup$
    – Outcast
    Commented May 30, 2019 at 18:03
  • $\begingroup$ Of course it will affect all the words, the goal is to give weight to the most relevant ones relatively to the least important ones. But yes you should test both options, it will probably confirm experimentally that TFIDF gives better results for this kind of case. $\endgroup$
    – Erwan
    Commented May 30, 2019 at 22:38
  • $\begingroup$ Ok let's see, thank you :) $\endgroup$
    – Outcast
    Commented May 31, 2019 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.