16
votes
Accepted
Can I use cosine similarity as a distance metric in a KNN algorithm
Short answer: Cosine distance is not the overall best performing distance metric out there
Although similarity measures are often expressed using a distance metric, it is in fact a more flexible ...
- 296
16
votes
Accepted
Why is the cosine distance used to measure the similatiry between word embeddings?
You're asking two questions here.
Does this mean the magnitude of the vectors is irrelevant?
Yes. Cosine similarity is $ S_{cos} = \frac{A \cdot B}{\|A\|\|B\|} $, which just comes from the ...
- 1,284
15
votes
Accepted
cosine_similarity returns matrix instead of single value
Based on the documentation cosine_similarity(X, Y=None, dense_output=True) returns an array with shape (n_samples_X, n_samples_Y). Your mistake is that you are ...
- 14.1k
13
votes
Accepted
Cosine similarity vs The Levenshtein distance
As mentioned in other answers, traditionally cosine is used to measure similarity between vectors whereas Levenshtein is used as a string similarity measure, i.e. measuring the distance between ...
- 25.7k
8
votes
Accepted
Calculate cosine similarity in Apache Spark
There's a related example to your problem in the Spark repo here. The strategy is to represent the documents as a RowMatrix and then use its columnSimilarities() method. That will get you a matrix ...
- 819
6
votes
Calculating cosine similarity between 3D arrays using Python
Your input matrices (with 3 rows and multiple columns) are saying that there are 3 samples, with multiple attributes. So the output you will get will be a 3x3 matrix, where each value is the ...
- 15k
5
votes
word2vec word embeddings creates very distant vectors, closest cosine similarity is still very far, only 0.7
Let us try and understand how Word2Vector actually works before looking at distances:
There are 2 ways of generating vectors for a word :
Continuous bag of words
Skip grams
The following diagram ...
- 785
5
votes
Cosine similarity vs The Levenshtein distance
I think the answers you got are technically correct, but don't address the big picture.
In the data science world, cosine similarity is mainly used for documents which have been encoded by an ...
- 326
5
votes
Accepted
Cosine similarity between sentence embeddings is always positive
Disclaimer: This is actually a tentative explanation, it provides a possible answer, but it does not contain proof.
First of all, contrary to added comments, cosine similarity is not always in the ...
- 4,773
4
votes
Accepted
How to find similar time series?
Since the time-series are annual, the data points you have for each time-series are limited and also quite distant (the values are 1 year apart). So I wouldn't use Dynamic Time Wrapping on your data.
...
- 576
4
votes
Accepted
Fastest way for 1 vs all lookup on embeddings
There are libraries that are specialized in exactly that task, for instance FAISS by Facebook AI Research:
Faiss is a library for efficient similarity search and clustering of dense vectors. It ...
- 27.2k
3
votes
cosine similarity between items (purchase data) and normalisation
My question is, do I need to normalize each product's vector before
using columnSimilarities()?
No, you do not need to normalize each product's vector before using columnSimilarities() since it is ...
- 73
3
votes
Accepted
Hierarchical clustering with precomputed cosine similarity matrix using scikit learn produces error
According to sklearn's documentation:
If linkage is “ward”, only “euclidean” is accepted. If
“precomputed”, a distance matrix (instead of a similarity matrix) is
needed as input for the fit ...
- 4,243
3
votes
Cosine similarity with arrays contaning NaN
I think it's rarely meaningful to consider cosine similarity on sparse data like this, not just because of sparsity (because it's only defined for dense data), but because it's not obvious the cosine ...
- 6,614
2
votes
How should I evaluate writing quality to compare two articles(which article is better suited/written for a topic ) according to their content?
"Source credibility" of Internet articles is best calculated through the Page Rank algorithm.
Algorithmically determining writing quality might be intractable. However Page Rank could be a proxy. If ...
- 22.1k
2
votes
Is it possible to use Jaccard similarity instead of Cosine similarity in gensim document similarity?
If you have trained a gensim model, that object can act as a dictionary to give you the vector projection (via https://radimrehurek.com/gensim/models/word2vec.html)
...
- 1,215
2
votes
memory error in matrix cosine_similarity
This is talking about RAM. There are a few factors that will decide how many rows/columns you can use. Instead of rows/columns, it is maybe easier to just think in total number of elements: ...
- 15k
2
votes
Approach to semantic similarity between documents
If I understand correctly, you're trying to map abstracts to their research papers.
Here is a simple starting point:
Compute a TF IDF model using the entire corpus (all abstracts + research papers). ...
- 467
2
votes
Accepted
Semantic similarity between two or more sentences
Word2vec as the name suggests will create an embedding for each word in your sentence. In order to get a sentence level embedding you would need to average (or combine in some other way) the ...
- 376
2
votes
If i use use BERT embeddings for if cosine(sent1,sent2) > 0.9, then is it fair to assume s1 and s2 are similar
They might or might not be similar, the embeddings extracted by mean pooling the BERT output usually have high cosine similarity even though the input sentences are completely different.
Bert ...
- 569
2
votes
Accepted
Cosine vs Manhattan for Text Similarity
Intuitively, if you normalized the vectors before using them, or if they all ended up having almost unit norm after training, then a small $l_1$ norm will imply that the angle between the vectors is ...
- 355
2
votes
String Matching
This problem is called record linkage. There are several related questions which might help:
About general techniques in record linkage
About Cosine vs. Levenshtein
About efficiency in record linkage:...
- 25.7k
2
votes
Semantic similarity on a large dataset
One approach would be to profile the code to empirically find the slowest parts. A quick visual scan of the code you referenced relieved inefficiencies.
For example, there are several list ...
- 22.1k
2
votes
Accepted
Convert cosine similarity to probability
You can try to normalize the similarity: norm_sim = (sim + 1) / 2, where sim is the cosine.
In this case, 0 means opposite similarity, 0.5 means no relationship ...
- 152
1
vote
Cosine similarity vs The Levenshtein distance
Cosine similarity uses vectors and can calculate similarity for sets and multisets (=bags). If used for similarity of sequences (of characters, words, sentences, lines, ...) the comparison is ...
1
vote
Cosine similarity vs The Levenshtein distance
The first one is for computing the similarity between objects considering their representations as vectors. The second one is for computing the similarity between sequences of characters.
1
vote
counter vector fit transform cosine similarity memory error
It's not clear to me what is your data and what you are trying to do with it, but from what I gather you are trying to calculate cosine similarity for each pair in a cartesian product, right?
If yes ...
- 25.7k
1
vote
Checking TF-IDF Results
Yes, Cosine TF-IDF is quite transparent so it's usually reasonably easy to visualize the words which contribute the most to a score. Cosine is defined as the dot product divided by the product of the ...
- 25.7k
1
vote
Accepted
Can I sum up feature vectors of a user‘s collection?
You can use total sum of boolean values. That will be fast and give a general notion of similarity.
A more useful metric might be Hamming distance, the sum of matching booleans between two vectors.
- 22.1k
1
vote
clustering 2-dimensional euclidean vectors - appropriate dissimilarity measure
For this kind of situation, spectral clustering is an intuitive solution. Basically, the idea is to perform the k-means clustering in a transformed feature space, by defining what the inner product ...
- 1,337
Only top scored, non community-wiki answers of a minimum length are eligible