39 votes
Accepted

Sentence similarity prediction

Your problem can be solved with Word2vec as well as Doc2vec. Doc2vec would give better results because it takes sentences into account while training the model. Doc2vec solution You can train your ...
  • 696
39 votes
Accepted

Adaboost vs Gradient Boosting

Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion. Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more ...
  • 6,135
36 votes

When to use cosine simlarity over Euclidean similarity

When to use cosine similarity over Euclidean similarity Cosine similarity looks at the angle between two vectors, euclidian similarity at the distance between two points. Let's say you are in an e-...
  • 18.3k
32 votes

Applications and differences for Jaccard similarity and Cosine Similarity

The answer from saq7 is wrong, as well as not answering the question. ∥A∥ means the $L2$ norm of $A$, i.e. the length of the vector in Euclidean space, not the dimensionality of the vector $A$. In ...
  • 421
20 votes
Accepted

Applications and differences for Jaccard similarity and Cosine Similarity

Jaccard Similarity is given by $s_{ij} = \frac{p}{p+q+r}$ where, p = # of attributes positive for both objects q = # of attributes 1 for i and 0 for j r = # of attributes 0 for i and 1 for j ...
  • 400
18 votes

Best practical algorithm for sentence similarity

Cosine Similarity for Vector Space could be you answer. Or you could calculate the eigenvector of each sentences. But the Problem is, what is similarity? "This is a tree", "This is not ...
17 votes
Accepted

How to measure the similarity between two images?

Check this handout! Well, there a few so... lets go: Given two images $J[x,y]$ and $I[x,y]$ with $(x,y) \in N^{N \times M}$... A - Used in template matching: Template Matching is linear and is not ...
14 votes
Accepted

Why use cosine similarity instead of scaling the vectors when calculating the similarity of vectors?

Let $u, v$ be vectors. The "cosine distance" between them is given by $$d_{\cos}(u, v) = 1 - \frac{u}{\|u\|} \cdot \frac{v}{\|v\|} = 1 - \cos \theta_{u,v},$$ and the proposed "...
12 votes

Applications and differences for Jaccard similarity and Cosine Similarity

Jaccard similarity is used for two types of binary cases: Symmetric, where 1 and 0 has equal importance (gender, marital status,etc) Asymmetric, where 1 and 0 have different levels of importance (...
12 votes

When to use cosine simlarity over Euclidean similarity

Ok, so, your intuition here is wrong. Not necessarily about the examples you gave, but the fact that you think Euclidian distance could be useful in 200 dimensional space. 200d space is so, so empty. ...
  • 241
11 votes
Accepted

Similarity measure based on multiple classes from a hierarchical taxonomy?

While I don't have enough expertise to advise you on selection of the best similarity measure, I've seen a number of them in various papers. The following collection of research papers hopefully will ...
10 votes

Sentence similarity prediction

Word Mover’s Distance (WMD) is an algorithm for finding the distance between sentences. WMD is based on word embeddings (e.g., word2vec) which encode the semantic meaning of words into dense vectors. ...
9 votes

Best practical algorithm for sentence similarity

One approach you could try is averaging word vectors generated by word embedding algorithms (word2vec, glove, etc). These algorithms create a vector for each word and the cosine similarity among them ...
  • 2,186
8 votes

Applications and differences for Jaccard similarity and Cosine Similarity

The answer by saq7 is wrong. Where $\mathbf{a}$ and $\mathbf{b}$ are binary vectors, they can be interpreted as sets of indices with value 1. Let's therefore consider sets $A$ and $B$. Jaccard ...
8 votes
Accepted

Text similarity with sentence embeddings

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector. The ...
8 votes
Accepted

Cosine similarity vs The Levenshtein distance

As mentioned in other answers, traditionally cosine is used to measure similarity between vectors whereas Levenshtein is used as a string similarity measure, i.e. measuring the distance between ...
  • 24.5k
7 votes
Accepted

How to measure the similarity between two text documents?

In general,there are two ways for finding document-document similarity TF-IDF approach Make a text corpus containing all words of documents . You have to use tokenisation and stop word removal . ...
7 votes

Similarity of words using BERTMODEL

First of all, I think you are confused with pretrained and finetuned. BERT is pretrained on ...
  • 949
5 votes
Accepted

Which supervised learning algorithms are available for matching?

You can try to frame this problem as a recommender systems situation. Where you have your users (prospective students) and items (alumni) and want to recommend to the users one item. It's not a ...
5 votes
Accepted

How to compute the Jaccard Similarity in this example? (Jaccard vs. Cosine)

Cosine similarity is for comparing two real-valued vectors, but Jaccard similarity is for comparing two binary vectors (sets). So you cannot compute the standard Jaccard similarity index between your ...
  • 3,860
5 votes

Sentence similarity prediction

You can try an easy solution using sklearn and it's going to work fine. Use tfidfvectorizer to get a vector representation of each text Fit the vectorizer with your data, removing stop-words. ...
5 votes

Calculate similarity on boolean data

You should look at the Jaccard Index, is the de facto similarity between set of items, where the sets are represented using a boolean vector. In this boolean vector each coordinate represents an item, ...
  • 2,186
5 votes

Why use cosine similarity instead of scaling the vectors when calculating the similarity of vectors?

In your example, User 1 and User 2 bought the same ingredients, but User 2 bought 100x more ingredients than User 1. If you normalize and use Euclidean distance, then the distance is 0 (by the ...
  • 629
4 votes

What is the difference between Latent and Explicit Semantic Analysis

The difference is that with ESA, the concepts are already known and labeled (hence, "manifest concepts"), whereas in LSA the concepts are latent (they are undefined and need to be discovered). Note ...
  • 836
4 votes

Best way to search for a similar document given the ngram

You could use a hashing vectorizer on your documents. The result will be a list of vectors. Then vectorize your ngrams in the same way and calculate the projection of this new vector on the old ones. ...
  • 550
4 votes

similarity measure with two features

Yes, multiple different ways. First, we could consider (id-artist,id-track) items as the elements of our set, and compute the Jaccard similarity by comparing those sets. Note that if the artist's id ...
4 votes

Vector space model cosine tf-idf for finding similar documents

Unfortunately, the math simplifies to show that you can't rigorously justify restricting the cosine similarity comparison of the vectors based on their lengths. The key point is that the cosine ...
  • 6,778
4 votes

How can I group texts with similar content together?

Well, after further googling I found the solution: MinHash or SimHash will do the job and I also found a tool implementing MinHash written in Scala on GitHub right at this link
  • 191
4 votes

Is there a way to measure correlation between two similar datasets?

I would take a look at Canonical correlation Analysis.
  • 1,307

Only top scored, non community-wiki answers of a minimum length are eligible