Questions tagged [cosine-distance]
A measure of the angular distance between two vectors. Usually defined as 1-(cosine similarity).
58
questions
0
votes
1answer
28 views
Clustering with TF-IDF and Cosine Similarity [closed]
I'm trying to cluster tf-idf vectors based on their cosine similarity, as such, I was experimenting with taking a given vector, calculating the mean cosine similarity with other vectors in the cluster,...
1
vote
0answers
25 views
Question about BERT embeddings with high cosine similarity
Under what circumstances would BERT assign two occurrences of the same word similar embeddings? If those occurrences are contained within similar syntactic relations with their co-occurrents?
0
votes
1answer
11 views
Semantic similarity between two or more sentences
I need to determine how similar sentences (in meaning) are to one another.
In order to do it, I have been considering an algorithm (cosine similarity) to determine the similarity between sentences. I ...
1
vote
1answer
27 views
Why is the cosine distance used to measure the similatiry between word embeddings?
While computing the similarity between the words, cosine similarity or distance is computed on word vectors. Why aren't other distance metrics such as Euclidean ...
1
vote
0answers
21 views
Is summing a cosine similarity matrix a good way to determine overall similarity?
I'm trying to similar research abstracts, so I'm using word embeddings to convert words into 1x768 vectors, so overall turning abstracts into embeddings with shape (#ofwords, 768). Cosine similarity ...
0
votes
0answers
30 views
How to compare cosine distances across two groups of words?
I am using a word2vec model based on Wikipedia corpus. I was looking for a way to quantify if two sets of words - s1= {a1, a2...}...
-1
votes
1answer
25 views
Matrix of pairwise cosine similarities from matrix of vectors [closed]
I have a matrix of 200d vectors stored as follows:
$ X = \begin{pmatrix} \text{id}_1 & 0.5 & -2 & \dots & 10
\\ \text{id}_2 & -4 & 6 & \dots & -0.3
...
1
vote
1answer
32 views
How to get the probability/closeness of a sample belonging to a specific cluster?
I'm new to this so please let me know if my logic of comparing cosine similarity and k-means is incorrect
I got a set of ...
0
votes
0answers
12 views
Should I create a tfidf on a subset of a data set or use the whole corpus?
My goal in this project is to see if businesses on a list are currently customers within my organization. One piece of this involves producing a similarity score using cosine similarity on the names ...
1
vote
3answers
196 views
Fastest way for 1 vs all lookup on embeddings
I have a dataset with about 1 000 000 texts where I have computed their sentence embeddings with a language model and stored them in a numpy array.
I wish to compare a new unseen text to all the 1 ...
0
votes
1answer
47 views
How fit_transform, transform and TfidfVectorizer works
I'm a machine learning beginner and I tried to use the cosine similarity on fuzzy matching purpose. In the following example I want to compare 'data_dirty' with 'data_clean' :
When I have to ...
1
vote
0answers
18 views
NearestNeighbors testing
I have used
nbrs = NearestNeighbors(metric= 'cosine', algorithm='brute').fit(items_features)
distances, indices = nbrs.kneighbors(item_features)
to find some ...
0
votes
0answers
92 views
Building Search Engine using Vector Space Model using a private database
Im trying to build a search engine for a private dataset using vector space model and have encountered following problem.
Dataset
Dataset is private. It is a collection of unstructed pdf .
I have ...
1
vote
1answer
51 views
Understanding cosine distance with word vectors
I'm a new DL4J user, and I'm running all the works of Shakespeare through a Word2Vec neural net. I've got a pretty basic question about how to understand the results so far. In the below example, ...
2
votes
1answer
58 views
Approach to semantic similarity between documents
I was wondering what approach people would take, or point me in the right direction on this challenge I have set myself. I am pretty new at this, I have covered some area but want to expand my ...
5
votes
1answer
71 views
Evaluating the performance of a machine learned recommendation system
I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes ...
0
votes
1answer
18 views
TensorFlow: CosineDifference ObjFunc Constant throughout training
The following example is a simplified version of what I'm working on. I'm trying to find a neural network which minimises the cosine distance. The reason I have implemented my own cosine difference ...
1
vote
3answers
1k views
Cosine similarity vs The Levenshtein distance
Cosine similarity vs The Levenshtein distance
I wanted to know what is the difference between them and in what situations they work best?
As per my understanding:
Cosine similarity is a measure of ...
0
votes
0answers
6 views
Implementation of the paper 'A Comprehensive Study for Center Loss'
I have studied the research paper A Comprehensive Study for Center Loss. The implementation in Caffe also exists in this github repo. In the paper, the author talks about a generalized implementation ...
0
votes
1answer
57 views
Siamese networks vs Semantic similarity (may be gensim)
I am trying to understand the Siamese networks . In this vector is calculated for an object (say an image) and a distance metric is applied (say manhatten) on two vectors produced by the neural ...
-1
votes
1answer
290 views
Word Embeddings with TFIDF vectorizer
I am a beginner in machine learning.
I have a large corpus of texts, divided into thematic groups. I would like to get idf values for the whole corpus, and then apply it on each group before ...
0
votes
1answer
211 views
counter vector fit transform cosine similarity memory error
count_matrix = count.fit_transform(off_data3['bag_of_words'])
I have count_matrix shape with
count_matrix.shape
(476147, 482824)
...
2
votes
1answer
2k views
memory error in matrix cosine_similarity
I have (20905040, 7)
of a dataset to recommend 10 different product to the user
it could be larger than that but anyway I got memory error when processing the
...
1
vote
1answer
293 views
Elbow method for cosine distance
I have clustered vectors by cosine distance using nltk clusterer. If I understand correctly, Y axis for elbow method in euclidian distance would be the sum of every distance (squared) between centroid ...
0
votes
1answer
26 views
Document matching with more priority to certain features than others
I am working on recommendation systems wherein I need to match the similarity of 2 users. Now, I know that I can use Tfidf vectorizer to calculate the the cosine similarity score between them. But, ...
4
votes
1answer
2k views
Calculating cosine similarity between 3D arrays using Python
I have two matrices with multiple columns and three rows each. I calculated the cosine similarity (sklearn) but it gives the result as a matrix. How can I obtain one single value?
The matrices are the ...
1
vote
1answer
47 views
Checking TF-IDF Results
I am working with TF-IDF and cosine similarity to do document comparisons and given a document, which document in the data is the most similar. However, sometimes it returns a high similarity between ...
2
votes
1answer
491 views
word2vec word embeddings creates very distant vectors, closest similarity is still very far
I started using gensim's FastText to create word embeddings on a large corpus of a specialized domain (after finding that existing open source embeddings are not performing well on this domain), ...
1
vote
1answer
3k views
Hierarchical clustering with precomputed cosine similarity matrix using scikit learn produces error
We want to use cosine similarity with hierarchical clustering and we have cosine similarities already calculated.
In the sklearn.cluster.AgglomerativeClustering documentation it says:
A distance ...
2
votes
3answers
717 views
Cosine similarity with arrays contaning NaN
I am trying to calculate a cosine similarity using Python in order to find similar users basing on ratings they have given to movies. As it can be expected there are a lot of NaN values.
I am using ...
0
votes
2answers
3k views
how to create word2vec for phrases and then calculate cosine similarity
I have just started using word2vec and I have no idea how to create vectors (using word2vec) of two lists, each containing set of words and phrases and then how to calculate cosine similarity between ...
2
votes
3answers
66 views
Match a two items from two different receipts
I have two different invoices or receipts. One is a Purchase order one is something like a receipt(acknowledgement).
Suppose I have ordered(PO) Wine:
White Wine
Red Wine
Rose Wine
And I receive ...
2
votes
1answer
28 views
When I would use a specific similarity coefficient over another?
Like using Jaccards over Dice.
I want real examples, of when I would prefer to use Jaccards, Dice, Cosine or any other similarity coefficient.
2
votes
1answer
781 views
cosine similarity between items (purchase data) and normalisation
I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. ...
1
vote
0answers
26 views
Best way to find dissimilarity in a 6x2 DataFrame?
I'm new to data science and am currently learning different techniques that I can do with Python. Currently, I'm trying it out with Spotify's API for my own playlists.
The goal is to find the most ...
1
vote
0answers
24 views
Can I sum up feature vectors of a userās collection?
I want to find items that are similar to items users already have in their collection. Every item has attributes, so I created feature vectors where every element of the vector represents an attribute ...
3
votes
3answers
549 views
clustering 2-dimensional euclidean vectors - appropriate dissimilarity measure
I've got a set of approx. 50 000 2-dimensional euclidean vectors which are connected with 20 groups, i.e. each group has approx. 2500 2-dimensional euclidean vectors. My data includes endpoints ...
2
votes
0answers
1k views
Clustering with cosine similarity with specific threshold (in Python)
Given :
1, either given the origin data matrix X , such as X = [[1,1,1,1],[1,1,-1,-1],[1,1,1,-1]]
or given the cosine similarity matrix of the original data X , ...
3
votes
1answer
3k views
How to find similar time series?
I've got a collection of yearly data (one value per year per category), and I'd like to find series that are most similar to one another.
Example data is here.
I don't know much about data science, ...
8
votes
1answer
21k views
cosine_similarity returns matrix instead of single value
I am using below code to compute cosine similarity between the 2 vectors. It returns a matrix instead of a single value 0.8660254.
[[ 1. 0.8660254]
[...
9
votes
4answers
10k views
Can I use cosine similarity as a distance metric in a KNN algorithm
Most discussions of KNN mention Euclidean,Manhattan and Hamming distances, but they dont mention cosine similarity metric. Is there a reason for this?
2
votes
2answers
599 views
Confusion with cosine similarity
In information retrieval when we calculate the cosine similarity between the query features vector and the document features vector we penalize the unseen words in the query.
Example if we have two ...
1
vote
1answer
4k views
Cosine similarity between query and document confusion
I am going through the Manning book for Information retrieval. Currently I am at the part about cosine similarity. One thing is not clear for me.
Let's say that I have the tf idf vectors for the ...
1
vote
1answer
4k views
Create similarity matrix
I have a training set and a testing set of vectors. All the vectors are labeled.
For each labeled vector in the testing set, there are 3 vectors in the training set with the same label.
I'm using ...
1
vote
1answer
978 views
Computing Item-to-Item Similarity Using Cosine
I have a "User x Item" matrix as below:
user item1 item2 item3
u1 2 0 3
u2 1 2 0
u3 4 3 1
u4 0 2 2
I want to computer the ...
1
vote
2answers
1k views
What techniques should I use to compare the similarity between a bunch of texts?
If I have a list of job postings stored as raw texts and I want to compare the similarity of all the job postings to a given resume, what techniques or algorithms should I use?
I'm thinking of ...
4
votes
1answer
179 views
How should I evaluate writing quality to compare two articles(which article is better suited/written for a topic ) according to their content?
I am trying to come up with a platform which can Synthesize Quality Content among many articles found about some particular "topic" on the internet, the algorithm should be able to say recommend top ...
2
votes
0answers
311 views
Recommendation matrix as a product of User Similarity and Ratings
For both item-item and user-user collaborative filtering the recommendation matrix $Ī_{m x n}$, which is an (m x n) matrix, can be defined as:
$$Ī(i,j)=r_{ij}$$
For user-user collaborative ...
13
votes
4answers
7k views
Alternatives to TF-IDF and Cosine Similarity when comparing documents of differing formats
I've been working on a small, personal project which takes a user's job skills and suggests the most ideal career for them based on those skills. I use a database of job listings to achieve this. At ...
2
votes
1answer
1k views
Is it possible to use Jaccard similarity instead of Cosine similarity in gensim document similarity?
I am using gensim library to compute similarity between documents but it only uses cosine similarity. I was wondering if there was a way to use jaccard similarity or any other similarity measure for ...
