Questions tagged [cosine-distance]

A measure of the angular distance between two vectors. Usually defined as 1-(cosine similarity).

Filter by
Sorted by
Tagged with
1 vote
0 answers
108 views

How to split graph data into train and test sets for link prediction problem using node emebbdings and cosine similarity

I would like to predict new links using node embeddings and cosine similarity, but I am unsure how to split the data set into training and testing, and how to evaluate new links. This is my code ...
Ghada Kahlaoui's user avatar
0 votes
0 answers
74 views

Siamese Network in TensorFlow employing Triplet Loss

I am constructing a siamese network using tensorflow which uses triplet loss. My inputs are of shape (100,100,1) and I have made a CNN embed_model to so that the output is a tensor with 50 points. Now ...
Adveat Prasad Karnik's user avatar
0 votes
0 answers
51 views

Unexpected cosine similarity score

The cosine similarity method has been working for my other cases. However, it returns anti-intuitive results for the following example, i.e. intuitively, I expect l2 has high score than l1, but ...
AnQ's user avatar
  • 1
0 votes
0 answers
84 views

On which texts should TfidfVectorizer be fitted when using TF-IDF cosine for text similarity?

I wonder on which texts should TfidfVectorizer be fitted when using TF-IDF cosine for text similarity. Should TfidfVectorizer be fitted on the texts that are analyzed for text similarity, or some ...
Franck Dernoncourt's user avatar
0 votes
0 answers
643 views

Using pyspark to create a large precomputed cosine similarity matrix from text data

I would like to precompute a cosine similarity matrix for a large dataset (upwards of 5 million rows) using pyspark. Here's what I have so far. libraries: ...
cfrench's user avatar
0 votes
0 answers
14 views

Numerical instabilety with kmeans

If i understand the math right a kmeans iteration should always improve cosine similarity. So if the data is z-normalized it should always improve corelation Well it seemed to be the case for a small ...
Nevo Krien's user avatar
0 votes
0 answers
25 views

For Q&A NLP system, how to extract the most relevant embedding if it is a combination of top K embeddings?

From my understanding, a typical "AI" Q&A system has a (vector) database of embedded text (from a set of documents). And when a user asks a question, the user's question is embedded and ...
siddgood's user avatar
1 vote
2 answers
1k views

Convert cosine similarity to probability

In natural language processing, the cosine similarity is often used to compute the similarity between two words. It is bounded between [-1, 1]. Supposedly, 1 means complete similarity, -1 means ...
postnubilaphoebus's user avatar
0 votes
0 answers
21 views

Applying feedback in content based recommendation

I have a content based recommender system, which finds similar items given a list of past liked items using cosine similarity. What would be best way to implement feedback or personalization in the ...
Kei Shuri's user avatar
0 votes
0 answers
71 views

Any heuristic to find the vector where cosine similarity will be maximum?

I want to find the sentence from the training data, which matches most with the prediction query text. You can assume, I have already generated the vector embeddings for each of the examples of the ...
hafiz031's user avatar
  • 121
0 votes
1 answer
174 views

Determine how each element in a vector contributes to cosine similarity when compare with other vector

I have a vector that represents my object and does a job of calculating which object is similar to the other object by using cosine similarity. To create that vector, I've combined many features that ...
Can Nguyen's user avatar
2 votes
2 answers
666 views

Semantic similarity on a large dataset

I'm going through this guide on semantic similarity and use the code there as is. I'm applying it to a dataset where each row is typically a paragraph (3-4 sentences, over 100 words). Currently, I ...
narrativera's user avatar
1 vote
1 answer
69 views

Personalized Recommendations In Content Based Recommendation System

I'm trying to create a content based recommender system. The system accuracy is quite enough when finding similar items but it's not as good as when recommending items to a specific user. I use ...
Ertugrul's user avatar
0 votes
1 answer
71 views

Converting images in a directory into a vector to calculate cosine distances?

I'm currently going through issues in terms of acquiring multiple images at once to convert them to a vector for calculating the cosine distance to get similarity between say an image from the ...
Is land's user avatar
  • 11
0 votes
0 answers
34 views

Why is there a 0.5 in this loss?

I'm reading this paper and I don't understand why the squared L2-norm is also multiplied by 0.5 in the loss. They want a loss that measures the distance between two feature maps. Why don't they use ...
Alessandro Polidori's user avatar
2 votes
0 answers
335 views

Cosine-like alternative to Mahalanobis distance

I would like to have a distance measure that takes into account how spread are vectors in a dataset, to weight the absolute distance from one point to another. The Mahalanobis distance does exactly ...
a_gdevr's user avatar
  • 21
1 vote
1 answer
51 views

Document Similarity with User Preference

To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
JoyfulPanda's user avatar
1 vote
2 answers
862 views

Can siamese model trained with euclidean distance as distance metric use cosine similarity during inference?

If I have 3 embeddings Anchor, Positive, Negative from a Siamese model trained with Euclidean distance as distance metric for triplet loss. During inference can ...
B200011011's user avatar
1 vote
2 answers
160 views

NLP Interview Coding Task

Please comment on the following NLP Interview Coding Task that I have prepared for the candidates on Data Science NLP position that I am looking for. The goal is to check candidate understanding of ...
dokondr's user avatar
  • 295
0 votes
0 answers
123 views

Conceptual question about cosine similarity and clustering algorithms for word embeddings

Is the following statement true? https://stats.stackexchange.com/q/256778 The value of cosine similarity between two terms itself is not indicator whether they are similar or not. If yes then how is ...
sigma_factor's user avatar
0 votes
0 answers
26 views

Which approach is beneficent for identifying the fake news detection?

The problem is to identify the fake news detection, As this is text classification problem . Constraints are basically that we cannot use traditional machine learning and deep learning approaches. If ...
Hamza's user avatar
  • 229
1 vote
1 answer
116 views

K-means++ with cosine distance

I am wondering how to implement k-means++ with cosine distance, acording to quote below (wikipedia), which says, that distance needs to be squared. But with square is lost direction of distance which ...
Night bird's user avatar
4 votes
1 answer
3k views

Cosine similarity between sentence embeddings is always positive

I have a list of documents and I am looking for a) duplicates; b) documents that are very similar. To do so, I proceed as follows: Embed the documents using paraphrase-xlm-r-multilingual-v1. ...
albus_c's user avatar
  • 177
1 vote
0 answers
122 views

Plotting cosine similarities in 3d space from word embeddings

I'm working on a project in which I want to estimate biases from a large corpus of newspaper articles using word2vec. Following this and this paper, biases are calculated by constructing dimension x ...
fritsvegters's user avatar
1 vote
0 answers
10 views

How to compute similarity matrix for strings efficiently? [duplicate]

Here I'm trying to compute similarity between 1000 cross 10000 strings (using Levenshtein distance), I'm using a dataframe approach where you just need to compare n(n-1)/2 comparisons instead of n*n. ...
David Gladson's user avatar
0 votes
1 answer
22 views

applicability of relative similarity computation

I've computed the cosine similarity between a & b (=x) and ...
Van Peer's user avatar
  • 285
1 vote
1 answer
385 views

String Matching [closed]

I have to work on 2 datasets where I have to find out the duplicates in addresses present in both the dataset. I am a bit confused that which one of the Levenshtein distance or cosine similarity, I ...
Debi Prasad's user avatar
0 votes
1 answer
347 views

Cosine vs Manhattan for Text Similarity [closed]

I'm storing sentences in Elasticsearch as dense_vector field and used BERT for the embedding so each vector is 768 dim. ...
Mohy Mohamed's user avatar
1 vote
0 answers
164 views

Matching documents from different sets with tfidf and cosine distance

I have two different set of documents S1, S2, with 30 text documents each. Using some text representation method, such as tfidf ...
lynx's user avatar
  • 113
1 vote
0 answers
207 views

how to calculate the cosine similarity between two files?

I am using spark and scala to implement an issue. files contain phrases or sentences. I want to use domain based method to calculate the cosine similarity between tags.I convert two files into a ...
salm's user avatar
  • 11
1 vote
1 answer
1k views

pairwise_distances with Cosine and weighting

Is there a way to get a weight into the pairwise_distances(X, metric='cosine') Potentially using **kwrds? ...
Rolfey's user avatar
  • 11
1 vote
1 answer
2k views

Calculate the similarity between pairs of time series data

I have 5 pieces of time series data. It is the weekly sales of 5 different stores (A,B,C,D,E). There are no missing values. A quick visual inspection shows that these 5 pieces of time series data have ...
DPatrick's user avatar
  • 113
2 votes
1 answer
391 views

Cosine Similarity but with weighting for vector indexes

I am very new to NLP and although this seems like a basic question I don't know how to search for an answer online. This is my problem: I have extracted and ranked keywords from 2 text sources: A ...
K Kreid's user avatar
  • 21
1 vote
2 answers
204 views

Evaluate document similarity / content-based recommender system

I'm planning on building a basic content-based recommender system with word2vec and cosine similarity. The data consists of 300k documents in varying length. How do I evaluate my model if I have no ...
jonas's user avatar
  • 143
1 vote
1 answer
155 views

finding similarity of a new datapoint

I have built a recommendation engine using cosine similarity. When I want to find all the records similar to a given record that is already present in the dataset it works. Consider a case, a user ...
Raj's user avatar
  • 159
1 vote
1 answer
61 views

Item-to-Item recommendation using DNN

I am new to DNN still learning, have a need to build item-to-item content based recommendation using DNN. For example, say I have a column of strings where each row represents a document I need to ...
Raj's user avatar
  • 159
1 vote
1 answer
869 views

If i use use BERT embeddings for if cosine(sent1,sent2) > 0.9, then is it fair to assume s1 and s2 are similar

According to BERT author Jacob Devlin: I'm not sure what these vectors are, since BERT does not generate meaningful sentence vectors. It seems that this is doing average pooling over the word tokens ...
user2478236's user avatar
2 votes
1 answer
323 views

Question about BERT embeddings with high cosine similarity

Under what circumstances would BERT assign two occurrences of the same word similar embeddings? If those occurrences are contained within similar syntactic relations with their co-occurrents?
Gog's user avatar
  • 23
0 votes
1 answer
2k views

Semantic similarity between two or more sentences

I need to determine how similar sentences (in meaning) are to one another. In order to do it, I have been considering an algorithm (cosine similarity) to determine the similarity between sentences. I ...
Math's user avatar
  • 151
9 votes
1 answer
7k views

Why is the cosine distance used to measure the similatiry between word embeddings?

While computing the similarity between the words, cosine similarity or distance is computed on word vectors. Why aren't other distance metrics such as Euclidean ...
Ashwin Geet D'Sa's user avatar
2 votes
2 answers
853 views

Is summing a cosine similarity matrix a good way to determine overall similarity?

I'm trying to similar research abstracts, so I'm using word embeddings to convert words into 1x768 vectors, so overall turning abstracts into embeddings with shape (#ofwords, 768). Cosine similarity ...
Brian Guan's user avatar
-1 votes
1 answer
1k views

Matrix of pairwise cosine similarities from matrix of vectors [closed]

I have a matrix of 200d vectors stored as follows: $ X = \begin{pmatrix} \text{id}_1 & 0.5 & -2 & \dots & 10 \\ \text{id}_2 & -4 & 6 & \dots & -0.3 ...
user11128's user avatar
2 votes
2 answers
881 views

How to get the probability/closeness of a sample belonging to a specific cluster?

I'm new to this so please let me know if my logic of comparing cosine similarity and k-means is incorrect I got a set of ...
Jaskaran Singh Puri's user avatar
1 vote
0 answers
30 views

Should I create a tfidf on a subset of a data set or use the whole corpus?

My goal in this project is to see if businesses on a list are currently customers within my organization. One piece of this involves producing a similarity score using cosine similarity on the names ...
Ian's user avatar
  • 111
1 vote
3 answers
2k views

Fastest way for 1 vs all lookup on embeddings

I have a dataset with about 1 000 000 texts where I have computed their sentence embeddings with a language model and stored them in a numpy array. I wish to compare a new unseen text to all the 1 ...
Isbister's user avatar
  • 173
2 votes
1 answer
1k views

How fit_transform, transform and TfidfVectorizer works

I'm a machine learning beginner and I tried to use the cosine similarity on fuzzy matching purpose. In the following example I want to compare 'data_dirty' with 'data_clean' : When I have to ...
nananinanana's user avatar
1 vote
0 answers
24 views

NearestNeighbors testing

I have used nbrs = NearestNeighbors(metric= 'cosine', algorithm='brute').fit(items_features) distances, indices = nbrs.kneighbors(item_features) to find some ...
AFB's user avatar
  • 41
1 vote
1 answer
304 views

Understanding cosine distance with word vectors

I'm a new DL4J user, and I'm running all the works of Shakespeare through a Word2Vec neural net. I've got a pretty basic question about how to understand the results so far. In the below example, ...
DPCII's user avatar
  • 35
2 votes
1 answer
197 views

Approach to semantic similarity between documents

I was wondering what approach people would take, or point me in the right direction on this challenge I have set myself. I am pretty new at this, I have covered some area but want to expand my ...
user5067291's user avatar
6 votes
1 answer
117 views

Evaluating the performance of a machine learned recommendation system

I have a set of resumes $R=\{{r_1,...,r_n\}}$, which I've transformed to a vector space using TF-IDF. Each resume has a label, which is the name of their current employer. Each of these labels comes ...
Data's user avatar
  • 467