Questions tagged [similarity]
The similarity tag has no usage guidance.
268
questions
11
votes
3
answers
1k
views
Why use cosine similarity instead of scaling the vectors when calculating the similarity of vectors?
I'm watching a NLP video on Coursera. It's discussing how to calculate the similarity of two vectors. First it discusses calculating the Euclidean distance, then it discusses the cosine similarity. It ...
- 253
0
votes
1
answer
29
views
Adjusted Cosine Similarity With Zero Vectors
I create a recommendation engine which finds item similarities according to user ratings. I'm trying to use adjusted cosine similarity to find similarities. I follow these steps.
Find mean rating of ...
0
votes
0
answers
26
views
Check if two bar charts are similar
Given two bar charts, I want to check if they are "similar". By similarity I mean that they have same number of clusters with same centers. Or may be I should not use cluster terminology?
In ...
- 101
0
votes
0
answers
11
views
How do calculate the distance between the centers of these two points for a series of images, and the points between one image to another divergent?
How do I calculate the distance between the centers of these two points for a series of images, and the points between one image to another divergent?
- 1
0
votes
0
answers
18
views
Calculating similarity score between nodes with the significance of their indexes
I have two excel files :
In the first file, each row represent node number and its features as
232 1 0 0 1 0 1
477 1 1 0 0 0 0
In the second file, each ...
- 1
0
votes
1
answer
16
views
Compute similarity with given weights for each different feature
I would like to find similar products based on the features.
I have:
3 categorical features (X1,X2,X3)
1 numerical (continuous) feature (X4)
1 date feature (X5)
Therefore, I want to give a pre-...
- 101
0
votes
0
answers
18
views
Siamese Network constant accuracy with more training data
I am new to machine learning and I am currently trying to create a siamese network that can predict the similarity of brand logos.
I have a dataset with ~210.000 brand logos.
The CNN for the siamese ...
0
votes
0
answers
26
views
How To Match a User with Another User Based on Similar Spotify Music Taste?
I'm a software engineer with a background in applied math, but I'm not too familiar with Data Science so I was wondering if anyone could help me with my question.
I want to match a user with another ...
- 1
1
vote
1
answer
15
views
Recommendation engine optimization
Recommendation engines typically include propensities to like the items (e.g, in a content-based approach). However, they generally do not take into account the fact that the customer may buy the ...
- 133
0
votes
0
answers
11
views
Metric for correlation of two paired lists of numbers
I have a program which produces an image, and I use a metric to understand how accurate that image is. I choose five cases (A, B, C, D, E), and make a list of the accuracy metric for each case:
...
1
vote
1
answer
27
views
A way to perform voting and select a candidate based on nearest neighbours
I'm working on a project where I use FAISS to retrieve n neighbouring vectors based on a query vector. The data in question is textual and is being embedded by using a machine learning model to create ...
- 111
0
votes
1
answer
46
views
How to measure similarities between two datasets with same features?
I have multiple datasets with the same features, a few numerical and a few categorical. The only difference is that they are market behavior for different countries. I wanted to know if there is a way ...
- 103
0
votes
0
answers
6
views
Search one 2D distribution for point cluster most similar to another 2D distribution
Given a hand drawn constellation (2d distribution of points) and a map of all stars, how would you find the actual star distribution most similar to the drawn distribution?
If it's helpful, suppose we ...
- 184
0
votes
0
answers
10
views
How to cluster time series of ordered data?
There are a few hundred time series of a large set of different locations (irregularly distributed) with the following properties:
ordered factor (5 levels)
between 5 and 25 observations per series
...
- 1
1
vote
1
answer
98
views
Building a graph out of a large text corpus
I'm given a large amount of documents upon which I should perform various kinds of analysis. Since the documents are to be used as a foundation of a final product, I thought about building a graph out ...
1
vote
1
answer
22
views
What is the logic/algorithm behind 'did you mean' suggestion by search engines, command suggestion in command prompt like git?
For eg. https://stackoverflow.com/questions/307291/how-does-the-google-did-you-mean-algorithm-work
this is the logic behind google's did you mean algorithm - used for spell correction suggestion.
What ...
- 11
1
vote
1
answer
30
views
High Performance Classification or Similarity Algorithim for Mixed Data Types?
I have a database holding 10-ish features that describe different breeds of dogs. They are mostly categorical features, but some provide ranges for values. Here's a demo representation of the database,...
0
votes
0
answers
51
views
What ways can i find two similar sets of customers use KNN?
I have a study where i want to find users similar to a set of users (SEED).
My data looks like a pivot by customer e.g. sample of SEED looks like (note i drop cust_id):
...
- 486
0
votes
1
answer
26
views
Word similarity considering special characteristics
I'm looking for an algorithm that computes the similarity between two strings just like the levenshtein distance. However, I want to consider the following. The <...
- 1
1
vote
1
answer
347
views
Measuring similarity from massive embedded vectors
I am given a set of 10,000 journal articles, with their corresponding 100th-dimension embedded vectors. (The way they are embedded is unknown, but I'm guessing it is ...
- 13
0
votes
0
answers
14
views
How to generate a penalty matrix after KMeans clustering
Let say we have 9 observations that can be grouped into 3 groups using K-means clustering:
...
- 101
1
vote
1
answer
22
views
Document Similarity with User Preference
To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
- 111
0
votes
1
answer
55
views
How to measure the similarity between two medical images of different imaging modalities according to similar objects in both of them?
I have two series of medical images each one from different imaging modalities.
According to that, I have been segmented the Region of interest (the object which appears in both modalities )using U-...
- 1
0
votes
0
answers
115
views
Comparing two images for similarity
Comparing two images for similarity
What are the best softwares available for comparing two images having similarity and differences?
Example: Margaret Thatcher & Enid Blyton.
- 159
0
votes
1
answer
19
views
Transformer similarity fine-tuned way too often predicts pairs as similar
I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data:
...
- 101
0
votes
0
answers
56
views
How to find similar document (using gensim) given two or more other documents?
I am developing a similarity program to compare documents, and I’ve successfully trained my model with Gensim (TFIDF and LSI) in order to compare two documents of each other, and it works great. I can ...
- 101
0
votes
0
answers
165
views
How to choose similarity measurement between sentences and paragraphs
Problems
1. How to find appropriate measurement method
There are several ways to measure sentence similarities, but I have no idea how to find appropriate method among them for my data (sentences).
...
- 37
0
votes
0
answers
37
views
Reproducing research results - one-shot classification for MNIST and Fashion-MNIST
I have come across two pieces of research related to one-shot classification of MNIST and Fashion-MNIST images using a 1-nearest neighbour (1-NN) classifier:
[1] G. Koch, "Siamese Neural ...
- 1
0
votes
1
answer
48
views
Dynamic Time Warping (DTW) for time series with different step sizes
Is it possible to use Dynamic Time Warping (DTW) algorithm as a method to find the similarity between two time-series data that have different step/measurement sizes? For example, one is measured ...
- 117
0
votes
1
answer
78
views
Converting similarity value into a dissimilarity value
Suppose we have similarity values between some data point in the interval $[0, 1]$. How can I transform this similarity values into a dissimilarity values in the interval $[0, ∞]$?
- 121
4
votes
1
answer
157
views
Has bloom filters a higher probability of collide when strings are similar
Because if it is the case, this really suits my need.
I'm trying to discover popular searches on a website. For this, I'm using TopK algorithm which is based on Bloom Filter hashing.
I don't want &...
- 453
2
votes
0
answers
13
views
Measuring the distance between data points based on mutual linkages
How to measure the distance between two data points (or: nodes?) based on their mutual share of linkages?
I don't know the technical term for that, so here is a fictitious example from scientific ...
- 123
2
votes
1
answer
41
views
Efficiently finding most similar sports from large dataset
As part of a past interview task, I’m working with a sports streaming dataset that looks like this:
...
0
votes
0
answers
74
views
Fuzzy Name String Matching
I am in need of solving the below given requirement.
Requirement:
I have two datasets which has only one column called Name. That column contains a list of user names in both the datasets so from this ...
0
votes
0
answers
12
views
Calculate comparable values for semantic relations between phrases
I'm looking for the best approach to calculate a way of comparing semantic relationships between phrases. In the following extremely simplified example:
"White sharks live in oceans. All dogs ...
- 1
1
vote
0
answers
10
views
How to compute similarity matrix for strings efficiently? [duplicate]
Here I'm trying to compute similarity between 1000 cross 10000 strings (using Levenshtein distance), I'm using a dataframe approach where you just need to compare n(n-1)/2 comparisons instead of n*n. ...
- 111
0
votes
1
answer
19
views
applicability of relative similarity computation
I've computed the cosine similarity between a & b (=x) and ...
- 285
1
vote
1
answer
78
views
Why checking the distribution of data is needed before calculating Gower distance?
I read this article(Clustering datasets having both numerical and categorical variables) to learn how to perform clustering on datasets with not just numerical variables.
Before calculating the Gower ...
4
votes
1
answer
55
views
encoding of text data in NLP
I'm getting data using web scraping to create a dataset. I have a 'company' column that contains the names of the companies. I would like to encode this column but i don't know how to find the ...
- 43
1
vote
0
answers
116
views
Performing actual deduplication using LSH
I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
- 121
0
votes
0
answers
13
views
Reducing dataset before computing similarity matrix
I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate ...
1
vote
0
answers
19
views
Testing similarity scores?
I need to calculate similarity between different houses using a series of attributes/properties, and to do that I need to define a certain similarity or distance function. Is there a way to test the ...
0
votes
1
answer
131
views
how to calculate similarity between users based on movie ratings
Hi I am working on a movie recommendation system and I have to find alikeness between the main user and other users. For example, the main user watched 3 specific movies and rated them as 8,5,7. A ...
- 25
0
votes
0
answers
38
views
Effective way to find similarity between utterance(short text) and question(long text)
The challenge I have is a bunch of questions(long text) that are closely matching with an user utterance (short text). I have tried cosine similarity & Tf-iDF, BM25,Jaccard similarity, etc., but ...
0
votes
1
answer
29
views
I'm building a movie recommendation system based on genre, where the user will enter his choice and will receive recommendations based on his choice:
Column Name: "Genres" //This will contain the genres of movies
User genre: "Action Adventure"
I want to perform cosine similarity on this data, compare user genre with genres of ...
- 1
1
vote
1
answer
24
views
how to find the best candidate [closed]
I am doing some work about entity disambiguation.
Please suppose, there are some candidates for each entity, e.g. e1 has three candidates c1, c2, c3.
Each candidate has two values: v1 and v2
We know ...
- 11
0
votes
1
answer
178
views
best similarity measure for images with different angles
I want to compare different images (where the images are of the same setup but the angles with which the images are taken are different). I want to obtain some sort of similarity score. I tried using ...
- 181
0
votes
1
answer
87
views
Non-commutative distance formula
I am trying to find a distance formula or a method that can give the non-commutative distance between two points in a feature space.
Suppose there are two movies represented in an R^n feature space. ...
- 1
1
vote
1
answer
243
views
Similarity between 2 statistical distributions
Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in ...
- 71
2
votes
2
answers
153
views
Find 10 most similar data points from 100?
I am trying to solve a problem where I need to group battery cells that are most similar to each other to form a reliable battery pack. It seems like a clustering problem but I only need to find the ...
