11

Based on the documentation cosine_similarity(X, Y=None, dense_output=True) returns an array with shape (n_samples_X, n_samples_Y). Your mistake is that you are passing [vec1, vec2] as the first input to the method. Also your vectors should be numpy arrays: from sklearn.metrics.pairwise import cosine_similarity import numpy as np vec1 = np.array([[1,1,0,1,1]]...


10

The cosine distance formula is: And the formula used by the cosine function of the spatial class of scipy is: So, the actual cosine similarity metric is: -0.9998. So, it signifies complete dissimilarity.


9

Short answer: Cosine distance is not the overall best performing distance metric out there Although similarity measures are often expressed using a distance metric, it is in fact a more flexible measure as it is not required to be symmetric or fulfill the triangle inequality. Nevertheless, it is very common to use a proper distance metric like the Euclidian ...


8

There's a related example to your problem in the Spark repo here. The strategy is to represent the documents as a RowMatrix and then use its columnSimilarities() method. That will get you a matrix of all the cosine similarities. Extract the row which corresponds to your query document and sort. That will give the indices of the most-similar documents. ...


4

Your input matrices (with 3 rows and multiple columns) are saying that there are 3 samples, with multiple attributes. So the output you will get will be a 3x3 matrix, where each value is the similarity to one other sample (there are 3 x 3 = 9 such combinations) If you were to print out the pairwise similarities in sparse format, then it might look closer to ...


3

Similarity measures are subjective and so are they ways to combine them. You should decide what is your subjective definition of similarity and then find a way to combine them that fit your definition. In general, I like to reduce similarity problems into classification problems. Given the dataset of items you have, create a new dataset of item pairs. The ...


2

If you have trained a gensim model, that object can act as a dictionary to give you the vector projection (via https://radimrehurek.com/gensim/models/word2vec.html) $ model['computer'] # raw numpy vector of a word array([-0.00449447, -0.00310097, 0.02421786, ...], dtype=float32) So it is possible to manually implement any vector comparison that you ...


2

"Source credibility" of Internet articles is best calculated through the Page Rank algorithm. Algorithmically determining writing quality might be intractable. However Page Rank could be a proxy. If an article is a hub then it is the authority on the topic and can be assumed well written (or at least very useful).


2

Let us try and understand how Word2Vector actually works before looking at distances: There are 2 ways of generating vectors for a word : Continuous bag of words Skip grams The following diagram explains the difference between the two approaches. In case you want to further understand the nitty gritty of these two approaches, there are tons of blogs out ...


2

My question is, do I need to normalize each product's vector before using columnSimilarities()? No, you do not need to normalize each product's vector before using columnSimilarities() since it is performed within the operation already. I think your confusion comes from the fact that your considering dot product and cosine similarity to be the same. They ...


2

Since the time-series are annual, the data points you have for each time-series are limited and also quite distant (the values are 1 year apart). So I wouldn't use Dynamic Time Wrapping on your data. If you are interested in comparing the patterns, a very simple approach would be Pearson's correlation. Keep in mind that this will not compare the actual ...


2

As mentioned in other answers, traditionally cosine is used to measure similarity between vectors whereas Levenshtein is used as a string similarity measure, i.e. measuring the distance between sequences of characters. Nevertheless they both can be used in non-traditional settings and are indeed comparable: the vectors compared with cosine can for instance ...


2

If I understand correctly, you're trying to map abstracts to their research papers. Here is a simple starting point: Compute a TF IDF model using the entire corpus (all abstracts + research papers). Use this model to transform your abstracts and research papers into a weighted vector representation. Under the TF IDF weighting scheme, these documents will ...


2

There are libraries that are specialized in exactly that task, for instance FAISS by Facebook AI Research: Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and ...


1

@lsbister, You could create a pandas dataframe and use a dask function/lambda function to parellize the computation of one vs all at the same time. If you use dask, you can create partitions and map the response back. In case you use pandas, you can use the apply function and parallelize the computations to a certain extent.


1

To the extent possible you should try to evaluate based on your data rather than some ad-hoc measure. As you rightly noticed, there is a real risk that the ad-hoc measure would just confirm the predictions of the model, since it uses a somewhat similar method. I would suggest that you split your data between a training set and test set (or even better use ...


1

The first one is for computing the similarity between objects considering their representations as vectors. The second one is for computing the similarity between sequences of characters.


1

Yes, Cosine TF-IDF is quite transparent so it's usually reasonably easy to visualize the words which contribute the most to a score. Cosine is defined as the dot product divided by the product of the norms, so you can isolate the terms: dotproduct(d_1,d_2) = tfidf(w1,d1) * tfidf(w1,d2) + tfidf(w2,d1) * tfidf(w2,d2) + ... + tfidf(wN,dN) Ranking the words ...


1

If you are using Python try the fuzzywuzzy package: FuzzyWuzzy Fuzzy string matching like a boss. It uses Levenshtein Distance to calculate the differences between sequences in a simple-to-use package. (Source)


1

You can try some approximate string matching which gives a confidence score. For example, you can try out with Levenshtein distance, but adjusted with the length of the strings using a probabilistic model; or, you can try out with Jaccard similarity on 3-grams and special treatment on word boundaries, and then calibrated into probabilities. Now you have an n ...


1

Jaccard - measures similarity of assymetric, binary attributes. For example, if you have insurance claims with binary attributes ("poor driving record", "premium paid in cash") you can compare claims with those attributes. Cosine - measures similarity between vectors, like feature vectors. Could be used in a recommender system where a user asks to see items ...


1

as Lejafar mentioned cosine violates triangle inequality however maybe this repo will help you


1

A common and simple method to match "documents" is to use TF-IDF weighting, as you have described. However, as I understand your question, you want to rank each career (-document) based on a set of users skills. If you create a "query vector" from the skills, you can multiply the vector with your term-career matrix (with all the tf-idf weights as values). ...


1

Perhaps you could use word embeddings to better represent the distance between certain skills. For instance, "Python" and "R" should be closer together than "Python" and "Time management" since they are both programming languages. The whole idea is that words that appear in the same context should be closer. Once you have these embeddings, you would have a ...


1

You want to use all of the terms in the vector. In your example, where your query vector $\mathbf{q} = [0,1,0,1,1]$ and your document vector $\mathbf{d} = [1,1,1,0,0]$, the cosine similarity is computed as similarity $= \frac{\mathbf{q} \cdot \mathbf{d}}{||\mathbf{q}||_2 ||\mathbf{d}||_2} = \frac{0\times1+1\times1+0\times1+1\times0+1\times0}{\sqrt{1^2+1^2+...


1

Filtering users will create bias in your training data. This may be good or bad, depending on your data and goals. The best way to find out, for your specific system, is to try and test both methods, optimizing for whatever metric you choose is best. Honestly, I think you should consider all users. Say you have a system for recommending movies. In this ...


1

As of now, I can think of two ways to formulate this problem: 1. Search problem Parse your job listings and index them in some sort of search engine like Solr or ElasticSearch. You can build capabilities like Semantic search using Word2Vec models, etc. Now write a query engine which takes a resume and queries this Search engine. It will be blazing fast ...


1

Yes, they are the same. The array [1, 0, 1, 0] represents a vector in $4$ dimensional euclidean space ($\mathbb{R}^4$) with tails at [0, 0 , 0 ,0] and head at [1, 0, 1, 0]. So your vector is "pointing" in the direction of your given set of coordinates. This might be easier to visualize in $2$ dimensions: for example, the array [1,0] corresponds to the ...


1

As you ask specifically for the Cosine Similarity technique, it has magnitude and direction, and similar to a vector which is used in Physics, as Cosine Similarity deals with vectors in an inner product space. So, the magnitude of vectors is exactly the same as the formula in Physics (summating over the squares of the vector elements.)


1

Implementation-side, there is a good reason to make 0 correspond to not rated. Since most users haven't rated most books, 0 will be the most common value and the cosine similarity function can use sparse matrices internally to speed up the computation. The sparse matrix shortcut is the main reason why people use cosine similarity in the first place. On ...


Only top voted, non community-wiki answers of a minimum length are eligible