# tf-idf for sentence level features

Many papers mention comparing sentences using the tf-idf metric, e.g. Paper.

They state:

The first one is based on tf-idf where the value of the the corresponding dimension in the vector representation is the number of occurrences of the word in the sentence times the idf (inverse document frequency) of the word.

While I am familiar with tf-idf weights per token, it is a bit vague for me how to extract a similarity measure between two sentences given the tf-idf weights of their individual tokens.

If the reference to the paper itself was not clear, the questions is: Given a document containing several sentences,

Is there a known measure of similarity between sentences in the document, based on the tf-idf score of the tokens inside each sentence?

It's common to see some confusion about TFIDF so thank you for asking this question :)

## TFIDF is not a metric, it's a weighting scheme

This means that it's a way to represent a document, not to compare documents. TFIDF assumes a bag of words (BoW) representation, i.e. a document or sentence is represented as a set of words (their order doesn't matter). The basic BoW representation is to encode every token/word with its frequency (TF); In TFIDF the frequency of the word is multiplied by the IDF (actually the log of the IDF) in order to give more importance to words which appear rarely.

Two important points:

• The TF part is specific to the document, whereas the IDF part is calculated across all the documents in the collection.
• Each dimension in a TFIDF vector represents a word. The dimensions are the same for all the documents, they correspond to the full vocabulary across all the documents (this way index $$i$$ always corresponds to the same word $$w_i$$).

Note that there are other weighting schemes which can be used to represent documents as vectors, for example Okapi BM25.

## Cosine-TFIDF is a metric to compare TFIDF vectors

Once documents (or sentences) have been encoded as TFIDF vectors using the same vocabulary (same dimensions), these vectors can be used to calculate a similarity score between any pair of documents (or a document and a query encoded the same way).

The Cosine similarity measure is certainly the most common way to compare TFIDF vectors. It's so common that sometimes people omit to mention it or over-simplify the explanation by saying that they "compare documents with TFIDF" (this is technically incorrect).

Note that other similarity measures can be used as well with TFIDF vectors. Most other measures (e.g. Jaccard) tend to give similar results, they're not fundamentally different from Cosine.

• I really enjoyed reading your answer, thank you. While comparing the tf-idf vectors of documents is reseanable, it makes no sense to me to compare tf-idf vectors of sentences, as they are extremely sensitive to typos, synonyms, and more. Nevertheless, I believe you are correct, and this is what they meant by "comparing sentences using tf-idf". Jul 21, 2021 at 13:06