Questions tagged [similarity]
The similarity tag has no usage guidance.
257
questions
1
vote
1
answer
16
views
What is the logic/algorithm behind 'did you mean' suggestion by search engines, command suggestion in command prompt like git?
For eg. https://stackoverflow.com/questions/307291/how-does-the-google-did-you-mean-algorithm-work
this is the logic behind google's did you mean algorithm - used for spell correction suggestion.
What ...
- 11
1
vote
1
answer
22
views
High Performance Classification or Similarity Algorithim for Mixed Data Types?
I have a database holding 10-ish features that describe different breeds of dogs. They are mostly categorical features, but some provide ranges for values. Here's a demo representation of the database,...
0
votes
0
answers
19
views
What ways can i find two similar sets of customers use KNN?
I have a study where i want to find users similar to a set of users (SEED).
My data looks like a pivot by customer e.g. sample of SEED looks like (note i drop cust_id):
...
- 466
0
votes
1
answer
18
views
Word similarity considering special characteristics
I'm looking for an algorithm that computes the similarity between two strings just like the levenshtein distance. However, I want to consider the following. The <...
- 1
1
vote
1
answer
91
views
Measuring similarity from massive embedded vectors
I am given a set of 10,000 journal articles, with their corresponding 100th-dimension embedded vectors. (The way they are embedded is unknown, but I'm guessing it is ...
- 13
0
votes
0
answers
10
views
How to generate a penalty matrix after KMeans clustering
Let say we have 9 observations that can be grouped into 3 groups using K-means clustering:
...
- 101
1
vote
1
answer
14
views
Document Similarity with User Preference
To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
- 111
0
votes
1
answer
20
views
How to measure the similarity between two medical images of different imaging modalities according to similar objects in both of them?
I have two series of medical images each one from different imaging modalities.
According to that, I have been segmented the Region of interest (the object which appears in both modalities )using U-...
- 1
0
votes
0
answers
76
views
Comparing two images for similarity
Comparing two images for similarity
What are the best softwares available for comparing two images having similarity and differences?
Example: Margaret Thatcher & Enid Blyton.
- 159
0
votes
1
answer
18
views
Transformer similarity fine-tuned way too often predicts pairs as similar
I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data:
...
- 101
0
votes
0
answers
18
views
How to find similar document (using gensim) given two or more other documents?
I am developing a similarity program to compare documents, and I’ve successfully trained my model with Gensim (TFIDF and LSI) in order to compare two documents of each other, and it works great. I can ...
- 101
0
votes
0
answers
58
views
How to choose similarity measurement between sentences and paragraphs
Problems
1. How to find appropriate measurement method
There are several ways to measure sentence similarities, but I have no idea how to find appropriate method among them for my data (sentences).
...
- 37
0
votes
0
answers
28
views
Reproducing research results - one-shot classification for MNIST and Fashion-MNIST
I have come across two pieces of research related to one-shot classification of MNIST and Fashion-MNIST images using a 1-nearest neighbour (1-NN) classifier:
[1] G. Koch, "Siamese Neural ...
- 1
0
votes
1
answer
32
views
Dynamic Time Warping (DTW) for time series with different step sizes
Is it possible to use Dynamic Time Warping (DTW) algorithm as a method to find the similarity between two time-series data that have different step/measurement sizes? For example, one is measured ...
- 117
0
votes
1
answer
32
views
Converting similarity value into a dissimilarity value
Suppose we have similarity values between some data point in the interval $[0, 1]$. How can I transform this similarity values into a dissimilarity values in the interval $[0, ∞]$?
- 101
4
votes
1
answer
146
views
Has bloom filters a higher probability of collide when strings are similar
Because if it is the case, this really suits my need.
I'm trying to discover popular searches on a website. For this, I'm using TopK algorithm which is based on Bloom Filter hashing.
I don't want &...
- 443
2
votes
0
answers
10
views
Measuring the distance between data points based on mutual linkages
How to measure the distance between two data points (or: nodes?) based on their mutual share of linkages?
I don't know the technical term for that, so here is a fictitious example from scientific ...
- 123
2
votes
1
answer
38
views
Efficiently finding most similar sports from large dataset
As part of a past interview task, I’m working with a sports streaming dataset that looks like this:
...
0
votes
0
answers
33
views
Fuzzy Name String Matching
I am in need of solving the below given requirement.
Requirement:
I have two datasets which has only one column called Name. That column contains a list of user names in both the datasets so from this ...
0
votes
0
answers
9
views
Calculate comparable values for semantic relations between phrases
I'm looking for the best approach to calculate a way of comparing semantic relationships between phrases. In the following extremely simplified example:
"White sharks live in oceans. All dogs ...
- 1
1
vote
0
answers
10
views
How to compute similarity matrix for strings efficiently? [duplicate]
Here I'm trying to compute similarity between 1000 cross 10000 strings (using Levenshtein distance), I'm using a dataframe approach where you just need to compare n(n-1)/2 comparisons instead of n*n. ...
- 111
0
votes
1
answer
18
views
applicability of relative similarity computation
I've computed the cosine similarity between a & b (=x) and ...
- 285
0
votes
0
answers
10
views
Inner working behind combining two distance function as one function for similarity measure
I am comparing two images and for this I am testing various similarity function. For my case, Euclidean works much better than cosine(20% difference). However, I tried to combine two distance function ...
- 13
0
votes
0
answers
25
views
Instagram Profile Similarity Features
I want to find similar IG accounts in a semantic way(not demografic like fan count, language, country,...) and thought of the following features:
Post Text Similarity (Embeddings by SBERT, averaging ...
- 1
1
vote
1
answer
54
views
Why checking the distribution of data is needed before calculating Gower distance?
I read this article(Clustering datasets having both numerical and categorical variables) to learn how to perform clustering on datasets with not just numerical variables.
Before calculating the Gower ...
4
votes
1
answer
49
views
encoding of text data in NLP
I'm getting data using web scraping to create a dataset. I have a 'company' column that contains the names of the companies. I would like to encode this column but i don't know how to find the ...
- 43
1
vote
0
answers
91
views
Performing actual deduplication using LSH
I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
- 121
0
votes
0
answers
13
views
Reducing dataset before computing similarity matrix
I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate ...
0
votes
0
answers
14
views
Find minimal group of 'spanning items'
I have a group of similarity hashes which I use to detect similar items using
hamming_distance < 10.
Each time a new item is being detected by one of the item ...
- 107
1
vote
0
answers
18
views
Testing similarity scores?
I need to calculate similarity between different houses using a series of attributes/properties, and to do that I need to define a certain similarity or distance function. Is there a way to test the ...
0
votes
1
answer
75
views
how to calculate similarity between users based on movie ratings
Hi I am working on a movie recommendation system and I have to find alikeness between the main user and other users. For example, the main user watched 3 specific movies and rated them as 8,5,7. A ...
- 15
0
votes
0
answers
34
views
Effective way to find similarity between utterance(short text) and question(long text)
The challenge I have is a bunch of questions(long text) that are closely matching with an user utterance (short text). I have tried cosine similarity & Tf-iDF, BM25,Jaccard similarity, etc., but ...
0
votes
1
answer
26
views
I'm building a movie recommendation system based on genre, where the user will enter his choice and will receive recommendations based on his choice:
Column Name: "Genres" //This will contain the genres of movies
User genre: "Action Adventure"
I want to perform cosine similarity on this data, compare user genre with genres of ...
- 1
1
vote
1
answer
23
views
how to find the best candidate [closed]
I am doing some work about entity disambiguation.
Please suppose, there are some candidates for each entity, e.g. e1 has three candidates c1, c2, c3.
Each candidate has two values: v1 and v2
We know ...
- 11
0
votes
1
answer
114
views
best similarity measure for images with different angles
I want to compare different images (where the images are of the same setup but the angles with which the images are taken are different). I want to obtain some sort of similarity score. I tried using ...
- 181
0
votes
1
answer
68
views
Non-commutative distance formula
I am trying to find a distance formula or a method that can give the non-commutative distance between two points in a feature space.
Suppose there are two movies represented in an R^n feature space. ...
- 1
1
vote
1
answer
119
views
Similarity between 2 statistical distributions
Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in ...
- 71
2
votes
2
answers
76
views
Find 10 most similar data points from 100?
I am trying to solve a problem where I need to group battery cells that are most similar to each other to form a reliable battery pack. It seems like a clustering problem but I only need to find the ...
1
vote
1
answer
73
views
Why does an imbalanced data set badly effect distance measures like Mahalanobis?
I'm relatively new to data science and I am struggling to understand why the Mahalanobis distance (or any other distance measure) applied to an imbalanced data-set becomes inaccurate. I have a data ...
- 13
0
votes
1
answer
107
views
Siamese vs matching network for correct image category matching
I have to find the closest match between my image and bunch of already collected images of different classes in the folder. Whic meta-learning approach should I select. I am thinking about the Siamese ...
- 13
1
vote
1
answer
298
views
Hierarchical clustering with the consensus matrix as similarity matrix
I'm following this article on consensus clustering in Python programming. On page 7 the authors state that "The consensus matrix lends itself naturally to be used as a visualization tool to help ...
- 121
0
votes
1
answer
90
views
Similarity Matching Algorithm
I am looking for help on identifying a class of algorithm. If I have a tabular training and test set I want to know the similarity of rows based on some numeric features. The training data would be ...
- 326
1
vote
1
answer
44
views
How to find coherence between a large number of sentences
I have a list of sentences returned as a result of a document search algorithm. I want to determine if the results returned are semantically close/similar/coherent using some sort of metric. For a ...
- 43
3
votes
3
answers
1k
views
Dot product for similarity in word to vector computation in NLP
In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word.
Uo is word vector for outside word
Vc is word ...
- 130
1
vote
1
answer
77
views
group similar subjects and train only using them
I have a dataset with 5k subjects. It's a binary classification problem where I have 3000 positive and 2000 negative subjects.
Now to build a model, I don't like to train the usual way (where we build ...
- 2,129
1
vote
1
answer
637
views
Calculate the similarity between pairs of time series data
I have 5 pieces of time series data. It is the weekly sales of 5 different stores (A,B,C,D,E). There are no missing values. A quick visual inspection shows that these 5 pieces of time series data have ...
- 113
1
vote
1
answer
60
views
Scoring metric for recommendation system
I'm working on a project that involves building a news recommendation system. I've come as far as quantifying user interaction with different articles on the site into user's affinity towards atopic ...
- 11
3
votes
1
answer
182
views
Pearson correlation with data sets that have values on different scales [closed]
I have two datasets with which I want to do a Pearson correlation analysis. I have carried out the analysis which makes sense, however I want to be sure it is valid given that both datasets have ...
- 39
1
vote
1
answer
28
views
Similarity Threshold Standards
When using similarity measures (eg. Resnik Information Content, Cosine Similarity, etc.) for any type of data, are there any standard similarity thresholds that are used, or does it all depend on the ...
- 37
1
vote
2
answers
284
views
Similarity between two time series with different sampling frequency, different amplitude, and different lengths but taken from the same source?
I have two files with accelerator readings and I want to get some metric/ measurement to get the similarity between these two files. I have tried Pearson’s R coefficient, dtw distance, dtw score. ...
- 11