Cosine Similarity but with weighting for vector indexes

Question

I am very new to NLP and although this seems like a basic question I don't know how to search for an answer online.

This is my problem: I have extracted and ranked keywords from 2 text sources:

A rank of 1, means this keyword is more important than a keyword with rank 5. Some keywords may not exist in one text but do in the other. In this case, where the keyword does not exist, there is no rank, therefore, Nan.

What method do I need to use to extract the similarity between the keyword ranks? I want to find out how similar the 2 texts are based on what keywords it contains and the ranks of these keywords.

I have tried cosine similarity by removing rows that contain Nan values and then treating text1Rank and text2Rank as vectors like this:

the 2 columns are the vectors I pass into the cosine similarity formula.

However, I do not think that this method weighs higher-ranked keywords more than lower-ranked keywords. Am I correct in thinking this?

If so, what method should I use to compare the 2 sets of ranks of keywords?

Answer 1

Cosine similarity won't work very well because it's only based on whether the rank at position $i$ is the same in vector 1 and vector 2.

For instance the vectors [3,2,4,1,5] vs [2,3,5,1,4] will have very low similarity because 4 positions out of 5 are different, even though there are only two swaps between (2,3) and (4,5).

A much better way to measure the similarity between two rankings is Spearman Rank Correlation.

Note that (if I'm not mistaken) in this case you could also directly use Pearson correlation, since the numeric values are already ranks. Spearman just ranks the values before applying Pearson correlation. So normally the two will give you the same result.

Note also that this method won't give more weight to the top of the ranking than the bottom. I'm not aware of a measure which does this, other than defining a custom weighted measure.