51

There's a number of different ways of going about this depending on exactly how much semantic information you want to retain and how easy your documents are to tokenize (html documents would probably be pretty difficult to tokenize, but you could conceivably do something with tags and context.) Some of them have been mentioned by ffriend, and the paragraph ...


38

Your problem can be solved with Word2vec as well as Doc2vec. Doc2vec would give better results because it takes sentences into account while training the model. Doc2vec solution You can train your doc2vec model following this link. You may want to perform some pre-processing steps like removing all stop words (words like "the", "an", etc. that don't add ...


35

Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion. Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more weight on misclassified samples and less weight on correctly classified samples. The final prediction is a weighted average of all the weak learners, where more ...


31

The answer from saq7 is wrong, as well as not answering the question. ∥A∥ means the $L2$ norm of $A$, i.e. the length of the vector in Euclidean space, not the dimensionality of the vector $A$. In other words, you don't count the 0 bits, you add up only the 1 bits and take the square root. Sorry I don't have a real answer as to when you should use which ...


30

When to use cosine similarity over Euclidean similarity Cosine similarity looks at the angle between two vectors, euclidian similarity at the distance between two points. Let's say you are in an e-commerce setting and you want to compare users for product recommendations: User 1 bought 1x eggs, 1x flour and 1x sugar. User 2 bought 100x eggs, 100x flour ...


19

Jaccard Similarity is given by $s_{ij} = \frac{p}{p+q+r}$ where, p = # of attributes positive for both objects q = # of attributes 1 for i and 0 for j r = # of attributes 0 for i and 1 for j Whereas, cosine similarity = $\frac{A \cdot B}{\|A\|\|B\|}$ where A and B are object vectors. Simply put, in cases where the vectors A and B are comprised 0s and 1s ...


17

Cosine Similarity for Vector Space could be you answer. Or you could calculate the eigenvector of each sentences. But the Problem is, what is similarity? "This is a tree", "This is not a tree" If you want to check the semantic meaning of the sentence you will need a wordvector dataset. With the wordvector dataset you will able to check ...


14

Check this handout! Well, there a few so... lets go: Given two images $J[x,y]$ and $I[x,y]$ with $(x,y) \in N^{N \times M}$... A - Used in template matching: Template Matching is linear and is not invariant to rotation (actually not even robust to it) but it is pretty simple and robust to noise such as the ones in photography taken with low illumination. ...


12

Jaccard similarity is used for two types of binary cases: Symmetric, where 1 and 0 has equal importance (gender, marital status,etc) Asymmetric, where 1 and 0 have different levels of importance (testing positive for a disease) Cosine similarity is usually used in the context of text mining for comparing documents or emails. If the cosine similarity ...


11

I think a number of clustering algorithms that normally use a metric, do not actually rely on the metric properties (other than commutativity, but I think you'd have that here). For example, DBSCAN uses epsilon-neighborhoods around a point; there is nothing in there that specifically says the triangle inequality matters. So you can probably use DBSCAN, ...


10

You are doing the correct thing. Technically, this averaging leads to computing the centroid in the Euclidean space of a set of N points. The centroid works pretty well with cosine similarities (cosine of the angles between normalized vectors), e.g. the Rocchio algorithm.


10

Alex made a number of good points, though I might have to push back a bit on his implication that DBSCAN is the best clustering algorithm to use here. Depending on your implementation, and whether or not you're using accelerated indices (many implementations do not), your time and space complexity will both be O(n2), which is far from ideal. Personally, my ...


10

While I don't have enough expertise to advise you on selection of the best similarity measure, I've seen a number of them in various papers. The following collection of research papers hopefully will be useful to you in determining the optimal measure for your research. Please note that I intentionally included papers, using both frequentist and Bayesian ...


9

Word Mover’s Distance (WMD) is an algorithm for finding the distance between sentences. WMD is based on word embeddings (e.g., word2vec) which encode the semantic meaning of words into dense vectors. The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "...


9

One approach you could try is averaging word vectors generated by word embedding algorithms (word2vec, glove, etc). These algorithms create a vector for each word and the cosine similarity among them represents semantic similarity among the words. In the case of the average vectors among the sentences. A good starting point for knowing more about these ...


8

The answer by saq7 is wrong. Where $\mathbf{a}$ and $\mathbf{b}$ are binary vectors, they can be interpreted as sets of indices with value 1. Let's therefore consider sets $A$ and $B$. Jaccard similarity is then given by $$J(A, B) = \frac{|A \cap B|}{|A \cup B|} = \frac{|A \cap B|}{|A \cap B| + |A - B| + |B - A|}$$ Cosine similarity is then given by $$C(A,...


7

Ok, so, your intuition here is wrong. Not necessarily about the examples you gave, but the fact that you think Euclidian distance could be useful in 200 dimensional space. 200d space is so, so empty. Everything is far from everything else. That's why we use cosine similarity - because everything is far from everything else, so if two vectors are pointing in ...


7

In general,there are two ways for finding document-document similarity TF-IDF approach Make a text corpus containing all words of documents . You have to use tokenisation and stop word removal . NLTK library provides all . Convert the documents into tf-idf vectors . Find the cosine-similarity between them or any new document for similarity measure. You ...


7

First of all, I think you are confused with pretrained and finetuned. BERT is pretrained on a lot of text data. By using this pretrained BERT, you have a model that already have knowledge about text. BERT can then be finetuned on specific dataset, where BERT learn specific knowledge related to the dataset. That's why a finetuned BERT is bad on other ...


6

There's a number of semantic distance measures, each with its pros and cons. Here are just a few of them: cosine distance, inner product between document feature vectors; LSA, another vector-based model, but utilizing SVD for de-noising original term-document matrix; WordNet-based, human verified, though hardly extensible. Start with a simplest approach ...


6

Empirically I have found LSA vastly superior to LDA every time and on every dataset I have tried it on. I have talked to other people who have said the same thing. It's also been used to win a number of the SemEval competitions for measuring semantic similarity between documents, often in combinations with a wordnet based measure, so I wouldn't say it's ...


6

DBSCAN (see also: Generalized DBSCAN) does not require a distance. All it needs is a binary decision. Commonly, one would use "distance < epsilon" but nothing says you cannot use "similarity > epsilon" instead. Triangle inequality etc. are not required. Affinity propagation, as the name says, uses similarities. Hierarchical clustering, except for maybe ...


6

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector. The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to ...


5

Cosine similarity is for comparing two real-valued vectors, but Jaccard similarity is for comparing two binary vectors (sets). So you cannot compute the standard Jaccard similarity index between your two vectors, but there is a generalized version of the Jaccard index for real valued vectors which you can use in this case: $J_g(\Bbb{a}, \Bbb{b}) =\frac{\...


5

You can try an easy solution using sklearn and it's going to work fine. Use tfidfvectorizer to get a vector representation of each text Fit the vectorizer with your data, removing stop-words. Transform the new entry with the vectorizer previously trained Compute the cosine similarity between this representation and each representation of the elements in ...


5

You should look at the Jaccard Index, is the de facto similarity between set of items, where the sets are represented using a boolean vector. In this boolean vector each coordinate represents an item, 1 means the item is present, 0 otherwise. For example: for an universe of items banana, orange and apple. the set banana, orange will be represented by (1, 1, ...


5

As mentioned in other answers, traditionally cosine is used to measure similarity between vectors whereas Levenshtein is used as a string similarity measure, i.e. measuring the distance between sequences of characters. Nevertheless they both can be used in non-traditional settings and are indeed comparable: the vectors compared with cosine can for instance ...


4

Topological Data Analysis is a method explicitly designed for the setting you describe. Rather than a global distance metric, it relies only on a local metric of proximity or neighborhood. See: Topology and data and Extracting insights from the shape of complex data using topology. You can find additional resources at the website for Ayasdi.


4

As a naive solution I would suggest to first select the strings which contain the most frequent tokens inside the list. In this way you can get rid of irrelevant string. In the second phrase I would do a majority voting. Assuming the 3 sentences: Star Wars: Episode IV A New Hope | StarWars.com Star Wars Episode IV - A New Hope (1977) Star Wars: Episode IV -...


Only top voted, non community-wiki answers of a minimum length are eligible