39
votes
Accepted
Sentence similarity prediction
Your problem can be solved with Word2vec as well as Doc2vec. Doc2vec would give better results because it takes sentences into account while training the model.
Doc2vec solution
You can train your ...
- 696
39
votes
Accepted
Adaboost vs Gradient Boosting
Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion.
Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more ...
oW_♦
- 6,135
36
votes
When to use cosine simlarity over Euclidean similarity
When to use cosine similarity over Euclidean similarity
Cosine similarity looks at the angle between two vectors, euclidian similarity at the distance between two points.
Let's say you are in an e-...
- 18.3k
32
votes
Applications and differences for Jaccard similarity and Cosine Similarity
The answer from saq7 is wrong, as well as not answering the question.
∥A∥ means the $L2$ norm of $A$, i.e. the length of the vector in Euclidean
space, not the dimensionality of the vector $A$. In ...
- 421
20
votes
Accepted
Applications and differences for Jaccard similarity and Cosine Similarity
Jaccard Similarity is given by
$s_{ij} = \frac{p}{p+q+r}$
where,
p = # of attributes positive for both objects
q = # of attributes 1 for i and 0 for j
r = # of attributes 0 for i and 1 for j
...
- 400
18
votes
Best practical algorithm for sentence similarity
Cosine Similarity for Vector Space could be you answer.
Or you could calculate the eigenvector of each sentences. But the Problem is, what is similarity?
"This is a tree",
"This is not ...
- 351
17
votes
Accepted
How to measure the similarity between two images?
Check this handout!
Well, there a few so... lets go:
Given two images $J[x,y]$ and $I[x,y]$ with $(x,y) \in N^{N \times M}$...
A - Used in template matching:
Template Matching is linear and is not ...
- 1,586
14
votes
Accepted
Why use cosine similarity instead of scaling the vectors when calculating the similarity of vectors?
Let $u, v$ be vectors. The "cosine distance" between them is given by
$$d_{\cos}(u, v) = 1 - \frac{u}{\|u\|} \cdot \frac{v}{\|v\|} = 1 - \cos \theta_{u,v},$$
and the proposed "...
- 256
12
votes
Applications and differences for Jaccard similarity and Cosine Similarity
Jaccard similarity is used for two types of binary cases:
Symmetric, where 1 and 0 has equal importance (gender, marital status,etc)
Asymmetric, where 1 and 0 have different levels of importance (...
- 221
12
votes
When to use cosine simlarity over Euclidean similarity
Ok, so, your intuition here is wrong. Not necessarily about the examples you gave, but the fact that you think Euclidian distance could be useful in 200 dimensional space. 200d space is so, so empty. ...
- 241
11
votes
Accepted
Similarity measure based on multiple classes from a hierarchical taxonomy?
While I don't have enough expertise to advise you on selection of the best similarity measure, I've seen a number of them in various papers. The following collection of research papers hopefully will ...
- 6,518
10
votes
Sentence similarity prediction
Word Mover’s Distance (WMD) is an algorithm for finding the distance between sentences. WMD is based on word embeddings (e.g., word2vec) which encode the semantic meaning of words into dense vectors.
...
- 19.4k
9
votes
Best practical algorithm for sentence similarity
One approach you could try is averaging word vectors generated by word embedding algorithms (word2vec, glove, etc). These algorithms create a vector for each word and the cosine similarity among them ...
- 2,186
8
votes
Applications and differences for Jaccard similarity and Cosine Similarity
The answer by saq7 is wrong.
Where $\mathbf{a}$ and $\mathbf{b}$ are binary vectors, they can be interpreted as sets of indices with value 1. Let's therefore consider sets $A$ and $B$.
Jaccard ...
8
votes
Accepted
Text similarity with sentence embeddings
One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.
The ...
- 19.4k
8
votes
Accepted
Cosine similarity vs The Levenshtein distance
As mentioned in other answers, traditionally cosine is used to measure similarity between vectors whereas Levenshtein is used as a string similarity measure, i.e. measuring the distance between ...
- 24.5k
7
votes
Accepted
How to measure the similarity between two text documents?
In general,there are two ways for finding document-document similarity
TF-IDF approach
Make a text corpus containing all words of documents . You have to use tokenisation and stop word removal . ...
- 116
7
votes
Similarity of words using BERTMODEL
First of all, I think you are confused with pretrained and finetuned.
BERT is pretrained on ...
- 949
5
votes
Accepted
Which supervised learning algorithms are available for matching?
You can try to frame this problem as a recommender systems situation. Where you have your users (prospective students) and items (alumni) and want to recommend to the users one item.
It's not a ...
- 311
5
votes
Accepted
How to compute the Jaccard Similarity in this example? (Jaccard vs. Cosine)
Cosine similarity is for comparing two real-valued vectors, but Jaccard similarity is for comparing two binary vectors (sets). So you cannot compute the standard Jaccard similarity index between your ...
- 3,860
5
votes
Sentence similarity prediction
You can try an easy solution using sklearn and it's going to work fine.
Use tfidfvectorizer to get a vector representation of each text
Fit the vectorizer with your data, removing stop-words.
...
- 760
5
votes
Calculate similarity on boolean data
You should look at the Jaccard Index, is the de facto similarity between set of items, where the sets are represented using a boolean vector. In this boolean vector each coordinate represents an item, ...
- 2,186
5
votes
Why use cosine similarity instead of scaling the vectors when calculating the similarity of vectors?
In your example, User 1 and User 2 bought the same ingredients, but User 2 bought 100x more ingredients than User 1. If you normalize and use Euclidean distance, then the distance is 0 (by the ...
- 629
4
votes
What is the difference between Latent and Explicit Semantic Analysis
The difference is that with ESA, the concepts are already known and labeled (hence, "manifest concepts"), whereas in LSA the concepts are latent (they are undefined and need to be discovered).
Note ...
- 836
4
votes
Best way to search for a similar document given the ngram
You could use a hashing vectorizer on your documents. The result will be a list of vectors. Then vectorize your ngrams in the same way and calculate the projection of this new vector on the old ones. ...
- 550
4
votes
similarity measure with two features
Yes, multiple different ways.
First, we could consider (id-artist,id-track) items as the elements of our set, and compute the Jaccard similarity by comparing those sets. Note that if the artist's id ...
- 735
4
votes
Vector space model cosine tf-idf for finding similar documents
Unfortunately, the math simplifies to show that you can't rigorously justify restricting the cosine similarity comparison of the vectors based on their lengths.
The key point is that the cosine ...
- 6,778
4
votes
How can I group texts with similar content together?
Well, after further googling I found the solution: MinHash or SimHash will do the job and I also found a tool implementing MinHash written in Scala on GitHub right at this link
- 191
4
votes
Is there a way to measure correlation between two similar datasets?
I would take a look at Canonical correlation Analysis.
- 1,307
Only top scored, non community-wiki answers of a minimum length are eligible