15 votes
Accepted

Can I use cosine similarity as a distance metric in a KNN algorithm

Short answer: Cosine distance is not the overall best performing distance metric out there Although similarity measures are often expressed using a distance metric, it is in fact a more flexible ...
user avatar
  • 286
14 votes
Accepted

cosine_similarity returns matrix instead of single value

Based on the documentation cosine_similarity(X, Y=None, dense_output=True) returns an array with shape (n_samples_X, n_samples_Y). Your mistake is that you are ...
user avatar
  • 13.4k
10 votes
Accepted

Cosine Distance > 1 in scipy

The cosine distance formula is: And the formula used by the cosine function of the spatial class of scipy is: So, the ...
user avatar
  • 8,036
10 votes
Accepted

Why is the cosine distance used to measure the similatiry between word embeddings?

You're asking two questions here. Does this mean the magnitude of the vectors is irrelevant? Yes. Cosine distance is $ D_{cos} = \frac{A \cdot B}{\|A\|\|B\|} $, which just comes from the definition ...
user avatar
  • 1,154
8 votes
Accepted

Calculate cosine similarity in Apache Spark

There's a related example to your problem in the Spark repo here. The strategy is to represent the documents as a RowMatrix and then use its columnSimilarities() method. That will get you a matrix ...
user avatar
  • 754
6 votes

Calculating cosine similarity between 3D arrays using Python

Your input matrices (with 3 rows and multiple columns) are saying that there are 3 samples, with multiple attributes. So the output you will get will be a 3x3 matrix, where each value is the ...
user avatar
  • 14k
6 votes
Accepted

Cosine similarity vs The Levenshtein distance

As mentioned in other answers, traditionally cosine is used to measure similarity between vectors whereas Levenshtein is used as a string similarity measure, i.e. measuring the distance between ...
user avatar
  • 22.2k
5 votes

word2vec word embeddings creates very distant vectors, closest cosine similarity is still very far, only 0.7

Let us try and understand how Word2Vector actually works before looking at distances: There are 2 ways of generating vectors for a word : Continuous bag of words Skip grams The following diagram ...
user avatar
5 votes
Accepted

Cosine similarity between sentence embeddings is always positive

Disclaimer: This is actually a tentative explanation, it provides a possible answer, but it does not contain proof. First of all, contrary to added comments, cosine similarity is not always in the ...
user avatar
  • 4,428
4 votes
Accepted

How to find similar time series?

Since the time-series are annual, the data points you have for each time-series are limited and also quite distant (the values are 1 year apart). So I wouldn't use Dynamic Time Wrapping on your data. ...
user avatar
  • 470
3 votes

How to find similarity/distance matrix with mixed Continuous and Categorical data?

Similarity measures are subjective and so are they ways to combine them. You should decide what is your subjective definition of similarity and then find a way to combine them that fit your definition....
user avatar
  • 2,593
2 votes
Accepted

Is Vector in Cosine Similarity the same as vector in Physics?

As you ask specifically for the Cosine Similarity technique, it has magnitude and direction, and similar to a vector which is used in Physics, as Cosine Similarity deals with vectors in an inner ...
user avatar
  • 8,036
2 votes

How should I evaluate writing quality to compare two articles(which article is better suited/written for a topic ) according to their content?

"Source credibility" of Internet articles is best calculated through the Page Rank algorithm. Algorithmically determining writing quality might be intractable. However Page Rank could be a proxy. If ...
user avatar
2 votes

Is it possible to use Jaccard similarity instead of Cosine similarity in gensim document similarity?

If you have trained a gensim model, that object can act as a dictionary to give you the vector projection (via https://radimrehurek.com/gensim/models/word2vec.html) ...
user avatar
  • 1,205
2 votes

cosine similarity between items (purchase data) and normalisation

My question is, do I need to normalize each product's vector before using columnSimilarities()? No, you do not need to normalize each product's vector before using columnSimilarities() since it is ...
user avatar
  • 63
2 votes
Accepted

Hierarchical clustering with precomputed cosine similarity matrix using scikit learn produces error

According to sklearn's documentation: If linkage is “ward”, only “euclidean” is accepted. If “precomputed”, a distance matrix (instead of a similarity matrix) is needed as input for the fit ...
user avatar
2 votes

Cosine similarity with arrays contaning NaN

I think it's rarely meaningful to consider cosine similarity on sparse data like this, not just because of sparsity (because it's only defined for dense data), but because it's not obvious the cosine ...
user avatar
  • 6,415
2 votes

memory error in matrix cosine_similarity

This is talking about RAM. There are a few factors that will decide how many rows/columns you can use. Instead of rows/columns, it is maybe easier to just think in total number of elements: ...
user avatar
  • 14k
2 votes

Approach to semantic similarity between documents

If I understand correctly, you're trying to map abstracts to their research papers. Here is a simple starting point: Compute a TF IDF model using the entire corpus (all abstracts + research papers). ...
user avatar
  • 437
2 votes
Accepted

Fastest way for 1 vs all lookup on embeddings

There are libraries that are specialized in exactly that task, for instance FAISS by Facebook AI Research: Faiss is a library for efficient similarity search and clustering of dense vectors. It ...
user avatar
  • 16k
2 votes
Accepted

Semantic similarity between two or more sentences

Word2vec as the name suggests will create an embedding for each word in your sentence. In order to get a sentence level embedding you would need to average (or combine in some other way) the ...
user avatar
2 votes
Accepted

Cosine vs Manhattan for Text Similarity

Intuitively, if you normalized the vectors before using them, or if they all ended up having almost unit norm after training, then a small $l_1$ norm will imply that the angle between the vectors is ...
user avatar
  • 355
2 votes

String Matching

This problem is called record linkage. There are several related questions which might help: About general techniques in record linkage About Cosine vs. Levenshtein About efficiency in record linkage:...
user avatar
  • 22.2k
1 vote

Checking TF-IDF Results

Yes, Cosine TF-IDF is quite transparent so it's usually reasonably easy to visualize the words which contribute the most to a score. Cosine is defined as the dot product divided by the product of the ...
user avatar
  • 22.2k
1 vote

Match a two items from two different receipts

If you are using Python try the fuzzywuzzy package: FuzzyWuzzy Fuzzy string matching like a boss. It uses Levenshtein Distance to calculate the differences between sequences in a simple-to-use ...
user avatar
1 vote

Match a two items from two different receipts

You can try some approximate string matching which gives a confidence score. For example, you can try out with Levenshtein distance, but adjusted with the length of the strings using a probabilistic ...
user avatar
1 vote

When I would use a specific similarity coefficient over another?

Jaccard - measures similarity of assymetric, binary attributes. For example, if you have insurance claims with binary attributes ("poor driving record", "premium paid in cash") you can compare claims ...
user avatar
  • 662
1 vote

Can I sum up feature vectors of a user‘s collection?

You can use total sum of boolean values. That will be fast and give a general notion of similarity. A more useful metric might be Hamming distance, the sum of matching booleans between two vectors.
user avatar
1 vote

clustering 2-dimensional euclidean vectors - appropriate dissimilarity measure

For this kind of situation, spectral clustering is an intuitive solution. Basically, the idea is to perform the k-means clustering in a transformed feature space, by defining what the inner product ...
user avatar
1 vote

Can I use cosine similarity as a distance metric in a KNN algorithm

Although cosine similarity is not a proper distance metric as it fails the triangle inequality, it can be useful in KNN. However, be wary that the cosine similarity is greatest when the angle is the ...
user avatar
  • 111

Only top scored, non community-wiki answers of a minimum length are eligible