# Questions tagged [similarity]

The tag has no usage guidance.

257 questions
Filter by
Sorted by
Tagged with
1 vote
16 views

### What is the logic/algorithm behind 'did you mean' suggestion by search engines, command suggestion in command prompt like git?

For eg. https://stackoverflow.com/questions/307291/how-does-the-google-did-you-mean-algorithm-work this is the logic behind google's did you mean algorithm - used for spell correction suggestion. What ...
• 11
1 vote
22 views

### High Performance Classification or Similarity Algorithim for Mixed Data Types?

I have a database holding 10-ish features that describe different breeds of dogs. They are mostly categorical features, but some provide ranges for values. Here's a demo representation of the database,...
19 views

### What ways can i find two similar sets of customers use KNN?

I have a study where i want to find users similar to a set of users (SEED). My data looks like a pivot by customer e.g. sample of SEED looks like (note i drop cust_id): ...
• 466
18 views

### Word similarity considering special characteristics

I'm looking for an algorithm that computes the similarity between two strings just like the levenshtein distance. However, I want to consider the following. The <...
1 vote
91 views

### Measuring similarity from massive embedded vectors

I am given a set of 10,000 journal articles, with their corresponding 100th-dimension embedded vectors. (The way they are embedded is unknown, but I'm guessing it is ...
• 13
10 views

### How to generate a penalty matrix after KMeans clustering

Let say we have 9 observations that can be grouped into 3 groups using K-means clustering: ...
• 101
1 vote
14 views

### Document Similarity with User Preference

To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
• 111
20 views

### How to measure the similarity between two medical images of different imaging modalities according to similar objects in both of them?

I have two series of medical images each one from different imaging modalities. According to that, I have been segmented the Region of interest (the object which appears in both modalities )using U-...
76 views

### Comparing two images for similarity

Comparing two images for similarity What are the best softwares available for comparing two images having similarity and differences? Example: Margaret Thatcher & Enid Blyton.
18 views

### Transformer similarity fine-tuned way too often predicts pairs as similar

I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data: ...
• 101
18 views

### How to find similar document (using gensim) given two or more other documents?

I am developing a similarity program to compare documents, and I’ve successfully trained my model with Gensim (TFIDF and LSI) in order to compare two documents of each other, and it works great. I can ...
• 101
58 views

### How to choose similarity measurement between sentences and paragraphs

Problems 1. How to find appropriate measurement method There are several ways to measure sentence similarities, but I have no idea how to find appropriate method among them for my data (sentences). ...
• 37
28 views

### Reproducing research results - one-shot classification for MNIST and Fashion-MNIST

I have come across two pieces of research related to one-shot classification of MNIST and Fashion-MNIST images using a 1-nearest neighbour (1-NN) classifier: [1] G. Koch, "Siamese Neural ...
32 views

### Dynamic Time Warping (DTW) for time series with different step sizes

Is it possible to use Dynamic Time Warping (DTW) algorithm as a method to find the similarity between two time-series data that have different step/measurement sizes? For example, one is measured ...
• 117
32 views

### Converting similarity value into a dissimilarity value

Suppose we have similarity values between some data point in the interval $[0, 1]$. How can I transform this similarity values into a dissimilarity values in the interval $[0, ∞]$?
• 101
146 views

### Has bloom filters a higher probability of collide when strings are similar

Because if it is the case, this really suits my need. I'm trying to discover popular searches on a website. For this, I'm using TopK algorithm which is based on Bloom Filter hashing. I don't want &...
• 443
10 views

### Measuring the distance between data points based on mutual linkages

How to measure the distance between two data points (or: nodes?) based on their mutual share of linkages? I don't know the technical term for that, so here is a fictitious example from scientific ...
• 123
38 views

### Efficiently finding most similar sports from large dataset

As part of a past interview task, I’m working with a sports streaming dataset that looks like this: ...
33 views

### Fuzzy Name String Matching

I am in need of solving the below given requirement. Requirement: I have two datasets which has only one column called Name. That column contains a list of user names in both the datasets so from this ...
9 views

### Calculate comparable values for semantic relations between phrases

I'm looking for the best approach to calculate a way of comparing semantic relationships between phrases. In the following extremely simplified example: "White sharks live in oceans. All dogs ...
1 vote
10 views

### How to compute similarity matrix for strings efficiently? [duplicate]

Here I'm trying to compute similarity between 1000 cross 10000 strings (using Levenshtein distance), I'm using a dataframe approach where you just need to compare n(n-1)/2 comparisons instead of n*n. ...
18 views

### applicability of relative similarity computation

I've computed the cosine similarity between a & b (=x) and ...
• 285
10 views

### Inner working behind combining two distance function as one function for similarity measure

I am comparing two images and for this I am testing various similarity function. For my case, Euclidean works much better than cosine(20% difference). However, I tried to combine two distance function ...
25 views

### Instagram Profile Similarity Features

I want to find similar IG accounts in a semantic way(not demografic like fan count, language, country,...) and thought of the following features: Post Text Similarity (Embeddings by SBERT, averaging ...
• 1
1 vote
54 views

### Why checking the distribution of data is needed before calculating Gower distance?

I read this article(Clustering datasets having both numerical and categorical variables) to learn how to perform clustering on datasets with not just numerical variables. Before calculating the Gower ...
49 views

### encoding of text data in NLP

I'm getting data using web scraping to create a dataset. I have a 'company' column that contains the names of the companies. I would like to encode this column but i don't know how to find the ...
• 43
1 vote
91 views

### Performing actual deduplication using LSH

I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
13 views

### Reducing dataset before computing similarity matrix

I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate ...
14 views

### Find minimal group of 'spanning items'

I have a group of similarity hashes which I use to detect similar items using hamming_distance < 10. Each time a new item is being detected by one of the item ...
• 107
1 vote
18 views

### Testing similarity scores?

I need to calculate similarity between different houses using a series of attributes/properties, and to do that I need to define a certain similarity or distance function. Is there a way to test the ...
75 views

### how to calculate similarity between users based on movie ratings

Hi I am working on a movie recommendation system and I have to find alikeness between the main user and other users. For example, the main user watched 3 specific movies and rated them as 8,5,7. A ...
• 15
34 views

### Effective way to find similarity between utterance(short text) and question(long text)

The challenge I have is a bunch of questions(long text) that are closely matching with an user utterance (short text). I have tried cosine similarity & Tf-iDF, BM25,Jaccard similarity, etc., but ...
26 views

### I'm building a movie recommendation system based on genre, where the user will enter his choice and will receive recommendations based on his choice:

Column Name: "Genres" //This will contain the genres of movies User genre: "Action Adventure" I want to perform cosine similarity on this data, compare user genre with genres of ...
1 vote
23 views

### how to find the best candidate [closed]

I am doing some work about entity disambiguation. Please suppose, there are some candidates for each entity, e.g. e1 has three candidates c1, c2, c3. Each candidate has two values: v1 and v2 We know ...
• 11
114 views

### best similarity measure for images with different angles

I want to compare different images (where the images are of the same setup but the angles with which the images are taken are different). I want to obtain some sort of similarity score. I tried using ...
• 181
68 views

### Non-commutative distance formula

I am trying to find a distance formula or a method that can give the non-commutative distance between two points in a feature space. Suppose there are two movies represented in an R^n feature space. ...
1 vote
119 views

### Similarity between 2 statistical distributions

Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in ...
• 71
76 views

### Find 10 most similar data points from 100?

I am trying to solve a problem where I need to group battery cells that are most similar to each other to form a reliable battery pack. It seems like a clustering problem but I only need to find the ...
1 vote
73 views

### Why does an imbalanced data set badly effect distance measures like Mahalanobis?

I'm relatively new to data science and I am struggling to understand why the Mahalanobis distance (or any other distance measure) applied to an imbalanced data-set becomes inaccurate. I have a data ...
• 13
107 views

### Siamese vs matching network for correct image category matching

I have to find the closest match between my image and bunch of already collected images of different classes in the folder. Whic meta-learning approach should I select. I am thinking about the Siamese ...
1 vote
298 views

### Hierarchical clustering with the consensus matrix as similarity matrix

I'm following this article on consensus clustering in Python programming. On page 7 the authors state that "The consensus matrix lends itself naturally to be used as a visualization tool to help ...
• 121
90 views

### Similarity Matching Algorithm

I am looking for help on identifying a class of algorithm. If I have a tabular training and test set I want to know the similarity of rows based on some numeric features. The training data would be ...
• 326
1 vote
44 views

### How to find coherence between a large number of sentences

I have a list of sentences returned as a result of a document search algorithm. I want to determine if the results returned are semantically close/similar/coherent using some sort of metric. For a ...
1k views

### Dot product for similarity in word to vector computation in NLP

In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word. Uo is word vector for outside word Vc is word ...
• 130
1 vote
77 views

### group similar subjects and train only using them

I have a dataset with 5k subjects. It's a binary classification problem where I have 3000 positive and 2000 negative subjects. Now to build a model, I don't like to train the usual way (where we build ...
• 2,129
1 vote
637 views

### Calculate the similarity between pairs of time series data

I have 5 pieces of time series data. It is the weekly sales of 5 different stores (A,B,C,D,E). There are no missing values. A quick visual inspection shows that these 5 pieces of time series data have ...
• 113
1 vote
60 views

### Scoring metric for recommendation system

I'm working on a project that involves building a news recommendation system. I've come as far as quantifying user interaction with different articles on the site into user's affinity towards atopic ...
182 views

### Pearson correlation with data sets that have values on different scales [closed]

I have two datasets with which I want to do a Pearson correlation analysis. I have carried out the analysis which makes sense, however I want to be sure it is valid given that both datasets have ...
1 vote