4
$\begingroup$

A common way of calculating the cosine similarity between text based documents is to calculate tf-idf and then calculating the linear kernel of the tf-idf matrix.

TF-IDF matrix is calculated using TfidfVectorizer().

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(stop_words='english')
tfidf_matrix_content = tfidf.fit_transform(article_master['stemmed_content'])

Here article_master is a dataframe containing the text content of all the documents.
As explained by Chris Clark here, TfidfVectorizer produces normalised vectors; hence the linear_kernel results can be used as cosine similarity.

cosine_sim_content = linear_kernel(tfidf_matrix_content, tfidf_matrix_content)


This is where my confusion lies.

Effectively the cosine similarity between 2 vectors is:

InnerProduct(vec1,vec2) / (VectorSize(vec1) * VectorSize(vec2))

Linear kernel calculates the InnerProduct as stated here

Linear Kernel Formulae

So the questions are:

  1. Why am I not divding the inner product with the product of the magnitude of the vectors ?

  2. Why does the normalisation exempt me of this requirement ?

  3. Now if I wanted to calculate ts-ss similarity, could I still use the normalised tf-idf matrix and the cosine values (calculated by linear kernel only) ?

$\endgroup$
2
$\begingroup$

Normalised vectors have magnitude 1, so it doesn't matter if you explicitly divide by the magnitudes or not. It's mathematically equivalent either way.

I see no reason that you couldn't use normalised vectors in TS-SS, but it seems that the main motivation for using TS-SS in the first place is that it makes more sense for vectors that may have different magnitudes. I would try both cosine similarity and TS-SS for your problem, and see if there is a noticeable performance difference.

$\endgroup$
  • $\begingroup$ Aha! Now I get it: tf-idf vectoriser normalises the individual rows (vectors) so that they are all of length 1. Since cosine similarity is only concerned with the angle, the magnitude difference of the vectors does not matter. $\endgroup$ – kgkmeekg Oct 24 at 21:34
  • $\begingroup$ However the prime reason behind using ts-ss is to take into account both the angle and the difference in magnitude. Hence even though there is nothing wrong in using normalised vectors; however, that beats the whole purpose of using Triangle Similarity component. $\endgroup$ – kgkmeekg Oct 24 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.