3
$\begingroup$

I want to make my own stop words list, I computed tf-idf scores for my terms.

Can I consider those words highlighted with red to be stop word? and what should my threshold be for stop words that depend on tf-idf? Should I consider the high values of tf-idf as the most important words that I need to keep?

enter image description here

@Erwan answered this question, check their answer to the question they linked too it is very informative

Improve this question
$\endgroup$

1 Answer 1

Reset to default
8
$\begingroup$
  • There's no standard definition of stop-word, but in general stop words are very frequent words which don't contribute to the meaning of the text, like determiners, pronouns, etc. Importantly stop-word is a property which applies to unique words in the vocabulary. For example if the word $w$ is considered as a stop-word then this applies to all the occurrences of $w$ in the text, not only to some of them.
  • On the contrary TFIDF applies to the words in the sentences/documents, so the same word $w$ may have a different TFIDF value in different sentences/documents:
    • IDF is a property at the vocabulary level, i.e. all the occurrences of $w$ have the same IDF.
    • TF is specific to the sentence/document. If $w$ appears 3 times more often in document A than in document B, then it has 3 times higher TFIDF value in A than in B.

This is why it doesn't really make sense to consider the TFIDF value to select stop-words: the former is specific to a sentence/document but not the second. You could use the IDF part only, but there's no difference with just using the document frequency, and practically it would give the same results as using the overall frequency.

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ @FjkgB yes, if you want to optimize the system you can try different thresholds and pick the best one based on the performance. Note that in theory you should do this on a validation set, it's like hyper-parameter tuning (in practice it's not so important in this case). But it's also common to just decide an arbitrary threshold, for example defining every word which appears more than 100 times as stop word. Also it's not related but if you're going to do some classification task it's also useful to remove the least frequent words (it's often more useful than the most frequent words). $\endgroup$
    – Erwan
    Feb 17, 2022 at 13:33
  • $\begingroup$ I deleted my first comment by mistake, but thank you so much for the extra information too about removing the least frequent terms being more useful. $\endgroup$
    – FjkgB
    Feb 17, 2022 at 13:35
  • 1
    $\begingroup$ @FjkgB in case you want more detail about why it's useful to get rid of rare words you can have a look at this question. $\endgroup$
    – Erwan
    Feb 17, 2022 at 21:45
  • 1
    $\begingroup$ @FjkgB Happy to help :) $\endgroup$
    – Erwan
    Feb 18, 2022 at 23:32
  • 1
    $\begingroup$ @MehdiAbbassi You mean taking the average TFIDF across sentences for every unique word, right? yes this would work, but it's a technically complex way to do something simple: if one just counts the frequency of every unique word in the corpus, in general the highest frequency words are stop words and other words are content words. $\endgroup$
    – Erwan
    Feb 21 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.