Questions tagged [cosine-distance]
A measure of the angular distance between two vectors. Usually defined as 1-(cosine similarity).
73
questions
0
votes
0answers
15 views
Conceptual question about cosine similarity and clustering algorithms for word embeddings
Is the following statement true? https://stats.stackexchange.com/q/256778
The value of cosine similarity between two terms itself is not indicator whether they are similar or not.
If yes then how is ...
0
votes
0answers
22 views
Which approach is beneficent for identifying the fake news detection?
The problem is to identify the fake news detection, As this is text classification problem . Constraints are basically that we cannot use traditional machine learning and deep learning approaches.
If ...
1
vote
1answer
33 views
K-means++ with cosine distance
I am wondering how to implement k-means++ with cosine distance, acording to quote below (wikipedia), which says, that distance needs to be squared. But with square is lost direction of distance which ...
2
votes
1answer
87 views
Cosine similarity between sentence embeddings is always positive
I have a list of documents and I am looking for a) duplicates; b) documents that are very similar. To do so, I proceed as follows:
Embed the documents using paraphrase-xlm-r-multilingual-v1.
...
1
vote
0answers
33 views
Plotting cosine similarities in 3d space from word embeddings
I'm working on a project in which I want to estimate biases from a large corpus of newspaper articles using word2vec. Following this and this paper, biases are calculated by constructing dimension x ...
1
vote
0answers
10 views
How to compute similarity matrix for strings efficiently? [duplicate]
Here I'm trying to compute similarity between 1000 cross 10000 strings (using Levenshtein distance), I'm using a dataframe approach where you just need to compare n(n-1)/2 comparisons instead of n*n. ...
0
votes
1answer
18 views
applicability of relative similarity computation
I've computed the cosine similarity between a & b (=x) and ...
0
votes
0answers
9 views
Inner working behind combining two distance function as one function for similarity measure
I am comparing two images and for this I am testing various similarity function. For my case, Euclidean works much better than cosine(20% difference). However, I tried to combine two distance function ...
1
vote
1answer
30 views
String Matching [closed]
I have to work on 2 datasets where I have to find out the duplicates in addresses present in both the dataset.
I am a bit confused that which one of the Levenshtein distance or cosine similarity, I ...
0
votes
0answers
16 views
Results interpretation of AgglomerativeClustering labelling
First of all I would like to say that I'm quite new to python and even more new to scikit, and I'm also a self learner, so please forgive my banal question, but it doesn't look banal to me.
So, I have ...
0
votes
1answer
77 views
Cosine vs Manhattan for Text Similarity [closed]
I'm storing sentences in Elasticsearch as dense_vector field and used BERT for the embedding so each vector is 768 dim. ...
1
vote
0answers
52 views
Matching documents from different sets with tfidf and cosine distance
I have two different set of documents S1, S2, with 30 text documents each.
Using some text representation method, such as tfidf ...
1
vote
0answers
53 views
how to calculate the cosine similarity between two files?
I am using spark and scala to implement an issue. files contain phrases or sentences. I want to use domain based method to calculate the cosine similarity between tags.I convert two files into a ...
1
vote
1answer
141 views
pairwise_distances with Cosine and weighting
Is there a way to get a weight into the
pairwise_distances(X, metric='cosine')
Potentially using **kwrds?
...
1
vote
1answer
468 views
Calculate the similarity between pairs of time series data
I have 5 pieces of time series data. It is the weekly sales of 5 different stores (A,B,C,D,E). There are no missing values. A quick visual inspection shows that these 5 pieces of time series data have ...
2
votes
1answer
116 views
Cosine Similarity but with weighting for vector indexes
I am very new to NLP and although this seems like a basic question I don't know how to search for an answer online.
This is my problem: I have extracted and ranked keywords from 2 text sources:
A ...
1
vote
2answers
119 views
Evaluate document similarity / content-based recommender system
I'm planning on building a basic content-based recommender system with word2vec and cosine similarity.
The data consists of 300k documents in varying length.
How do I evaluate my model if I have no ...
1
vote
1answer
45 views
finding similarity of a new datapoint
I have built a recommendation engine using cosine similarity. When I want to find all the records similar to a given record that is already present in the dataset it works. Consider a case, a user ...
1
vote
1answer
41 views
Item-to-Item recommendation using DNN
I am new to DNN still learning, have a need to build item-to-item content based recommendation using DNN. For example, say I have a column of strings where each row represents a document I need to ...
1
vote
1answer
296 views
If i use use BERT embeddings for if cosine(sent1,sent2) > 0.9, then is it fair to assume s1 and s2 are similar
According to BERT author Jacob Devlin: I'm not sure what these vectors are, since BERT does not generate meaningful sentence vectors. It seems that this is doing average pooling over the word tokens ...
1
vote
0answers
77 views
Is the magnitude of a word vector correlated with the frequency of the word in a text?
In order to find the similarity between words, cosine similarity seems to be the most common measure to use. In a conversation that I had about this topic, someone mentioned that words that mean more ...
1
vote
1answer
194 views
Question about BERT embeddings with high cosine similarity
Under what circumstances would BERT assign two occurrences of the same word similar embeddings? If those occurrences are contained within similar syntactic relations with their co-occurrents?
0
votes
1answer
783 views
Semantic similarity between two or more sentences
I need to determine how similar sentences (in meaning) are to one another.
In order to do it, I have been considering an algorithm (cosine similarity) to determine the similarity between sentences. I ...
6
votes
1answer
2k views
Why is the cosine distance used to measure the similatiry between word embeddings?
While computing the similarity between the words, cosine similarity or distance is computed on word vectors. Why aren't other distance metrics such as Euclidean ...
1
vote
0answers
56 views
Is summing a cosine similarity matrix a good way to determine overall similarity?
I'm trying to similar research abstracts, so I'm using word embeddings to convert words into 1x768 vectors, so overall turning abstracts into embeddings with shape (#ofwords, 768). Cosine similarity ...
-1
votes
1answer
524 views
Matrix of pairwise cosine similarities from matrix of vectors [closed]
I have a matrix of 200d vectors stored as follows:
$ X = \begin{pmatrix} \text{id}_1 & 0.5 & -2 & \dots & 10
\\ \text{id}_2 & -4 & 6 & \dots & -0.3
...
2
votes
1answer
354 views
How to get the probability/closeness of a sample belonging to a specific cluster?
I'm new to this so please let me know if my logic of comparing cosine similarity and k-means is incorrect
I got a set of ...
1
vote
0answers
26 views
Should I create a tfidf on a subset of a data set or use the whole corpus?
My goal in this project is to see if businesses on a list are currently customers within my organization. One piece of this involves producing a similarity score using cosine similarity on the names ...
1
vote
3answers
1k views
Fastest way for 1 vs all lookup on embeddings
I have a dataset with about 1 000 000 texts where I have computed their sentence embeddings with a language model and stored them in a numpy array.
I wish to compare a new unseen text to all the 1 ...
0
votes
1answer
322 views
How fit_transform, transform and TfidfVectorizer works
I'm a machine learning beginner and I tried to use the cosine similarity on fuzzy matching purpose. In the following example I want to compare 'data_dirty' with 'data_clean' :
When I have to ...
1
vote
0answers
22 views
NearestNeighbors testing
I have used
nbrs = NearestNeighbors(metric= 'cosine', algorithm='brute').fit(items_features)
distances, indices = nbrs.kneighbors(item_features)
to find some ...
1
vote
1answer
130 views
Understanding cosine distance with word vectors
I'm a new DL4J user, and I'm running all the works of Shakespeare through a Word2Vec neural net. I've got a pretty basic question about how to understand the results so far. In the below example, ...
2
votes
1answer
118 views
Approach to semantic similarity between documents
I was wondering what approach people would take, or point me in the right direction on this challenge I have set myself. I am pretty new at this, I have covered some area but want to expand my ...
5
votes
1answer
87 views
Evaluating the performance of a machine learned recommendation system
I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes ...
0
votes
1answer
19 views
TensorFlow: CosineDifference ObjFunc Constant throughout training
The following example is a simplified version of what I'm working on. I'm trying to find a neural network which minimises the cosine distance. The reason I have implemented my own cosine difference ...
4
votes
4answers
4k views
Cosine similarity vs The Levenshtein distance
Cosine similarity vs The Levenshtein distance
I wanted to know what is the difference between them and in what situations they work best?
As per my understanding:
Cosine similarity is a measure of ...
0
votes
1answer
103 views
Siamese networks vs Semantic similarity (may be gensim)
I am trying to understand the Siamese networks . In this vector is calculated for an object (say an image) and a distance metric is applied (say manhatten) on two vectors produced by the neural ...
0
votes
1answer
390 views
counter vector fit transform cosine similarity memory error
count_matrix = count.fit_transform(off_data3['bag_of_words'])
I have count_matrix shape with
count_matrix.shape
(476147, 482824)
...
3
votes
1answer
5k views
memory error in matrix cosine_similarity
I have (20905040, 7)
of a dataset to recommend 10 different product to the user
it could be larger than that but anyway I got memory error when processing the
...
1
vote
1answer
567 views
Elbow method for cosine distance
I have clustered vectors by cosine distance using nltk clusterer. If I understand correctly, Y axis for elbow method in euclidian distance would be the sum of every distance (squared) between centroid ...
0
votes
1answer
53 views
Document matching with more priority to certain features than others
I am working on recommendation systems wherein I need to match the similarity of 2 users. Now, I know that I can use Tfidf vectorizer to calculate the the cosine similarity score between them. But, ...
5
votes
1answer
4k views
Calculating cosine similarity between 3D arrays using Python
I have two matrices with multiple columns and three rows each. I calculated the cosine similarity (sklearn) but it gives the result as a matrix. How can I obtain one single value?
The matrices are the ...
1
vote
1answer
67 views
Checking TF-IDF Results
I am working with TF-IDF and cosine similarity to do document comparisons and given a document, which document in the data is the most similar. However, sometimes it returns a high similarity between ...
4
votes
1answer
1k views
word2vec word embeddings creates very distant vectors, closest cosine similarity is still very far, only 0.7
I started using gensim's FastText to create word embeddings on a large corpus of a specialized domain (after finding that existing open source embeddings are not performing well on this domain), ...
1
vote
1answer
7k views
Hierarchical clustering with precomputed cosine similarity matrix using scikit learn produces error
We want to use cosine similarity with hierarchical clustering and we have cosine similarities already calculated.
In the sklearn.cluster.AgglomerativeClustering documentation it says:
A distance ...
3
votes
3answers
2k views
Cosine similarity with arrays contaning NaN
I am trying to calculate a cosine similarity using Python in order to find similar users basing on ratings they have given to movies. As it can be expected there are a lot of NaN values.
I am using ...
0
votes
2answers
4k views
How to create word2vec for phrases and then calculate cosine similarity
I just started using word2vec and have no idea how to create vectors (using word2vec) of two lists, each containing set of words and phrases and then how to calculate the cosine similarity between ...
2
votes
3answers
136 views
Match a two items from two different receipts
I have two different invoices or receipts. One is a Purchase order one is something like a receipt(acknowledgement).
Suppose I have ordered(PO) Wine:
White Wine
Red Wine
Rose Wine
And I receive ...
2
votes
1answer
71 views
When I would use a specific similarity coefficient over another?
Like using Jaccards over Dice.
I want real examples, of when I would prefer to use Jaccards, Dice, Cosine or any other similarity coefficient.
2
votes
1answer
2k views
cosine similarity between items (purchase data) and normalisation
I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. ...
