49

There's a number of different ways of going about this depending on exactly how much semantic information you want to retain and how easy your documents are to tokenize (html documents would probably be pretty difficult to tokenize, but you could conceivably do something with tags and context.) Some of them have been mentioned by ffriend, and the paragraph ...


27

The answer from saq7 is wrong, as well as not answering the question. ∥A∥ means the $L2$ norm of $A$, i.e. the length of the vector in Euclidean space, not the dimensionality of the vector $A$. In other words, you don't count the 0 bits, you add up only the 1 bits and take the square root. Sorry I don't have a real answer as to when you should use which ...


27

Your problem can be solved with Word2vec as well as Doc2vec. Doc2vec would give better results because it takes sentences into account while training the model. Doc2vec solution You can train your doc2vec model following this link. You may want to perform some pre-processing steps like removing all stop words (words like "the", "an", etc. that don't add ...


20

Both AdaBoost and Gradient Boosting build weak learners in a sequential fashion. Originally, AdaBoost was designed in such a way that at every step the sample distribution was adapted to put more weight on misclassified samples and less weight on correctly classified samples. The final prediction is a weighted average of all the weak learners, where more ...


19

Jaccard Similarity is given by $s_{ij} = \frac{p}{p+q+r}$ where, p = # of attributes positive for both objects q = # of attributes 1 for i and 0 for j r = # of attributes 0 for i and 1 for j Whereas, cosine similarity = $\frac{A \cdot B}{\|A\|\|B\|}$ where A and B are object vectors. Simply put, in cosine similarity, the number of common attributes ...


16

Cosine Similarity for Vector Space could be you answer: http://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii/ Or you could calculate the eigenvector of each sentences. But the Problem is, what is similarity? "This is a tree", "This is not a tree" If you want to check the semantic meaning of the ...


12

Jaccard similarity is used for two types of binary cases: Symmetric, where 1 and 0 has equal importance (gender, marital status,etc) Asymmetric, where 1 and 0 have different levels of importance (testing positive for a disease) Cosine similarity is usually used in the context of text mining for comparing documents or emails. If the cosine similarity ...


10

You are doing the correct thing. Technically, this averaging leads to computing the centroid in the Euclidean space of a set of N points. The centroid works pretty well with cosine similarities (cosine of the angles between normalized vectors), e.g. the Rocchio algorithm.


9

While I don't have enough expertise to advise you on selection of the best similarity measure, I've seen a number of them in various papers. The following collection of research papers hopefully will be useful to you in determining the optimal measure for your research. Please note that I intentionally included papers, using both frequentist and Bayesian ...


9

I think a number of clustering algorithms that normally use a metric, do not actually rely on the metric properties (other than commutativity, but I think you'd have that here). For example, DBSCAN uses epsilon-neighborhoods around a point; there is nothing in there that specifically says the triangle inequality matters. So you can probably use DBSCAN, ...


8

Alex made a number of good points, though I might have to push back a bit on his implication that DBSCAN is the best clustering algorithm to use here. Depending on your implementation, and whether or not you're using accelerated indices (many implementations do not), your time and space complexity will both be O(n2), which is far from ideal. Personally, my ...


8

One approach you could try is averaging word vectors generated by word embedding algorithms (word2vec, glove, etc). These algorithms create a vector for each word and the cosine similarity among them represents semantic similarity among the words. In the case of the average vectors among the sentences. A good starting point for knowing more about these ...


7

In general,there are two ways for finding document-document similarity TF-IDF approach Make a text corpus containing all words of documents . You have to use tokenisation and stop word removal . NLTK library provides all . Convert the documents into tf-idf vectors . Find the cosine-similarity between them or any new document for similarity measure. You ...


6

The answer by saq7 is wrong. Where $\mathbf{a}$ and $\mathbf{b}$ are binary vectors, they can be interpreted as sets of indices with value 1. Let's therefore consider sets $A$ and $B$. Jaccard similarity is then given by $$J(A, B) = \frac{|A \cap B|}{|A \cup B|} = \frac{|A \cap B|}{|A \cap B| + |A - B| + |B - A|}$$ Cosine similarity is then given by $$C(A,...


6

Empirically I have found LSA vastly superior to LDA every time and on every dataset I have tried it on. I have talked to other people who have said the same thing. It's also been used to win a number of the SemEval competitions for measuring semantic similarity between documents, often in combinations with a wordnet based measure, so I wouldn't say it's ...


6

There's a number of semantic distance measures, each with its pros and cons. Here are just a few of them: cosine distance, inner product between document feature vectors; LSA, another vector-based model, but utilizing SVD for de-noising original term-document matrix; WordNet-based, human verified, though hardly extensible. Start with a simplest approach ...


6

DBSCAN (see also: Generalized DBSCAN) does not require a distance. All it needs is a binary decision. Commonly, one would use "distance < epsilon" but nothing says you cannot use "similarity > epsilon" instead. Triangle inequality etc. are not required. Affinity propagation, as the name says, uses similarities. Hierarchical clustering, except for maybe ...


5

Cosine similarity is for comparing two real-valued vectors, but Jaccard similarity is for comparing two binary vectors (sets). So you cannot compute the standard Jaccard similarity index between your two vectors, but there is a generalized version of the Jaccard index for real valued vectors which you can use in this case: $J_g(\Bbb{a}, \Bbb{b}) =\frac{\...


5

You should look at the Jaccard Index, is the de facto similarity between set of items, where the sets are represented using a boolean vector. In this boolean vector each coordinate represents an item, 1 means the item is present, 0 otherwise. For example: for an universe of items banana, orange and apple. the set banana, orange will be represented by (1, 1, ...


5

Check this handout! Well, there a few so... lets go: Given two images $J[x,y]$ and $I[x,y]$ with $(x,y) \in N^{N \times M}$... A - Used in template matching: Template Matching is linear and is not invariant to rotation (actually not even robust to it) but it is pretty simple and robust to noise such as the ones in photography taken with low illumination. ...


4

Topological Data Analysis is a method explicitly designed for the setting you describe. Rather than a global distance metric, it relies only on a local metric of proximity or neighborhood. See: Topology and data and Extracting insights from the shape of complex data using topology. You can find additional resources at the website for Ayasdi.


4

One thing you could do is fuzzify your vectors: replace each 1 by (for example) 0.4 in its position, 0.2 in the neighbouring positions, and 0.1 in the second position over. Then add up what's in each position. With these fuzzified vectors, you can apply a similarity metric either based on a distance or one like cosines similarity. Your example would ...


4

As a naive solution I would suggest to first select the strings which contain the most frequent tokens inside the list. In this way you can get rid of irrelevant string. In the second phrase I would do a majority voting. Assuming the 3 sentences: Star Wars: Episode IV A New Hope | StarWars.com Star Wars Episode IV - A New Hope (1977) Star Wars: Episode IV -...


4

The difference is that with ESA, the concepts are already known and labeled (hence, "manifest concepts"), whereas in LSA the concepts are latent (they are undefined and need to be discovered). Note the following statement from the ESA Wikipedia page: The name "explicit semantic analysis" contrasts with latent semantic analysis (LSA), because the use of a ...


4

Yes, multiple different ways. First, we could consider (id-artist,id-track) items as the elements of our set, and compute the Jaccard similarity by comparing those sets. Note that if the artist's id gives us no additional information beyond the track id, this will give the same result, whereas it will give different results if a particular track id could be ...


4

Unfortunately, the math simplifies to show that you can't rigorously justify restricting the cosine similarity comparison of the vectors based on their lengths. The key point is that the cosine similarity metric normalizes based on length, so that only the unit vectors are considered. I know this isn't necessarily the answer that you wanted, but the math ...


4

I would take a look at Canonical correlation Analysis.


4

I see a lot of people post this similar question on StackExchange, and the truth is that there is no methodology to compare if data set A looks like set B. You can compare summary statistics, such as means, deviations, min/max, but there's no magical formula to say that data set A looks like B, especially if they are varying data sets by rows and columns. I ...


4

Word Mover’s Distance (WMD) is an algorithm for finding the distance between sentences. WMD is based on word embeddings (e.g., word2vec) which encode the semantic meaning of words into dense vectors. The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "...


Only top voted, non community-wiki answers of a minimum length are eligible