Questions tagged [similarity]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
16 views

What is the logic/algorithm behind 'did you mean' suggestion by search engines, command suggestion in command prompt like git?

For eg. https://stackoverflow.com/questions/307291/how-does-the-google-did-you-mean-algorithm-work this is the logic behind google's did you mean algorithm - used for spell correction suggestion. What ...
user avatar
  • 11
1 vote
1 answer
22 views

High Performance Classification or Similarity Algorithim for Mixed Data Types?

I have a database holding 10-ish features that describe different breeds of dogs. They are mostly categorical features, but some provide ranges for values. Here's a demo representation of the database,...
user avatar
0 votes
0 answers
19 views

What ways can i find two similar sets of customers use KNN?

I have a study where i want to find users similar to a set of users (SEED). My data looks like a pivot by customer e.g. sample of SEED looks like (note i drop cust_id): ...
user avatar
  • 466
0 votes
1 answer
18 views

Word similarity considering special characteristics

I'm looking for an algorithm that computes the similarity between two strings just like the levenshtein distance. However, I want to consider the following. The <...
user avatar
  • 1
1 vote
1 answer
91 views

Measuring similarity from massive embedded vectors

I am given a set of 10,000 journal articles, with their corresponding 100th-dimension embedded vectors. (The way they are embedded is unknown, but I'm guessing it is ...
user avatar
  • 13
0 votes
0 answers
10 views

How to generate a penalty matrix after KMeans clustering

Let say we have 9 observations that can be grouped into 3 groups using K-means clustering: ...
user avatar
  • 101
1 vote
1 answer
14 views

Document Similarity with User Preference

To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
user avatar
0 votes
1 answer
20 views

How to measure the similarity between two medical images of different imaging modalities according to similar objects in both of them?

I have two series of medical images each one from different imaging modalities. According to that, I have been segmented the Region of interest (the object which appears in both modalities )using U-...
user avatar
  • 1
0 votes
0 answers
76 views

Comparing two images for similarity

Comparing two images for similarity What are the best softwares available for comparing two images having similarity and differences? Example: Margaret Thatcher & Enid Blyton.
user avatar
0 votes
1 answer
18 views

Transformer similarity fine-tuned way too often predicts pairs as similar

I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data: ...
user avatar
  • 101
0 votes
0 answers
18 views

How to find similar document (using gensim) given two or more other documents?

I am developing a similarity program to compare documents, and I’ve successfully trained my model with Gensim (TFIDF and LSI) in order to compare two documents of each other, and it works great. I can ...
user avatar
  • 101
0 votes
0 answers
58 views

How to choose similarity measurement between sentences and paragraphs

Problems 1. How to find appropriate measurement method There are several ways to measure sentence similarities, but I have no idea how to find appropriate method among them for my data (sentences). ...
user avatar
  • 37
0 votes
0 answers
28 views

Reproducing research results - one-shot classification for MNIST and Fashion-MNIST

I have come across two pieces of research related to one-shot classification of MNIST and Fashion-MNIST images using a 1-nearest neighbour (1-NN) classifier: [1] G. Koch, "Siamese Neural ...
user avatar
  • 1
0 votes
1 answer
32 views

Dynamic Time Warping (DTW) for time series with different step sizes

Is it possible to use Dynamic Time Warping (DTW) algorithm as a method to find the similarity between two time-series data that have different step/measurement sizes? For example, one is measured ...
user avatar
  • 117
0 votes
1 answer
32 views

Converting similarity value into a dissimilarity value

Suppose we have similarity values between some data point in the interval $[0, 1]$. How can I transform this similarity values into a dissimilarity values in the interval $[0, ∞]$?
user avatar
  • 101
4 votes
1 answer
146 views

Has bloom filters a higher probability of collide when strings are similar

Because if it is the case, this really suits my need. I'm trying to discover popular searches on a website. For this, I'm using TopK algorithm which is based on Bloom Filter hashing. I don't want &...
user avatar
  • 443
2 votes
0 answers
10 views

Measuring the distance between data points based on mutual linkages

How to measure the distance between two data points (or: nodes?) based on their mutual share of linkages? I don't know the technical term for that, so here is a fictitious example from scientific ...
user avatar
  • 123
2 votes
1 answer
38 views

Efficiently finding most similar sports from large dataset

As part of a past interview task, I’m working with a sports streaming dataset that looks like this: ...
user avatar
0 votes
0 answers
33 views

Fuzzy Name String Matching

I am in need of solving the below given requirement. Requirement: I have two datasets which has only one column called Name. That column contains a list of user names in both the datasets so from this ...
user avatar
0 votes
0 answers
9 views

Calculate comparable values for semantic relations between phrases

I'm looking for the best approach to calculate a way of comparing semantic relationships between phrases. In the following extremely simplified example: "White sharks live in oceans. All dogs ...
user avatar
1 vote
0 answers
10 views

How to compute similarity matrix for strings efficiently? [duplicate]

Here I'm trying to compute similarity between 1000 cross 10000 strings (using Levenshtein distance), I'm using a dataframe approach where you just need to compare n(n-1)/2 comparisons instead of n*n. ...
user avatar
0 votes
1 answer
18 views

applicability of relative similarity computation

I've computed the cosine similarity between a & b (=x) and ...
user avatar
  • 285
0 votes
0 answers
10 views

Inner working behind combining two distance function as one function for similarity measure

I am comparing two images and for this I am testing various similarity function. For my case, Euclidean works much better than cosine(20% difference). However, I tried to combine two distance function ...
user avatar
0 votes
0 answers
25 views

Instagram Profile Similarity Features

I want to find similar IG accounts in a semantic way(not demografic like fan count, language, country,...) and thought of the following features: Post Text Similarity (Embeddings by SBERT, averaging ...
user avatar
  • 1
1 vote
1 answer
54 views

Why checking the distribution of data is needed before calculating Gower distance?

I read this article(Clustering datasets having both numerical and categorical variables) to learn how to perform clustering on datasets with not just numerical variables. Before calculating the Gower ...
user avatar
4 votes
1 answer
49 views

encoding of text data in NLP

I'm getting data using web scraping to create a dataset. I have a 'company' column that contains the names of the companies. I would like to encode this column but i don't know how to find the ...
user avatar
  • 43
1 vote
0 answers
91 views

Performing actual deduplication using LSH

I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
user avatar
0 votes
0 answers
13 views

Reducing dataset before computing similarity matrix

I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate ...
user avatar
0 votes
0 answers
14 views

Find minimal group of 'spanning items'

I have a group of similarity hashes which I use to detect similar items using hamming_distance < 10. Each time a new item is being detected by one of the item ...
user avatar
  • 107
1 vote
0 answers
18 views

Testing similarity scores?

I need to calculate similarity between different houses using a series of attributes/properties, and to do that I need to define a certain similarity or distance function. Is there a way to test the ...
user avatar
0 votes
1 answer
75 views

how to calculate similarity between users based on movie ratings

Hi I am working on a movie recommendation system and I have to find alikeness between the main user and other users. For example, the main user watched 3 specific movies and rated them as 8,5,7. A ...
user avatar
  • 15
0 votes
0 answers
34 views

Effective way to find similarity between utterance(short text) and question(long text)

The challenge I have is a bunch of questions(long text) that are closely matching with an user utterance (short text). I have tried cosine similarity & Tf-iDF, BM25,Jaccard similarity, etc., but ...
user avatar
0 votes
1 answer
26 views

I'm building a movie recommendation system based on genre, where the user will enter his choice and will receive recommendations based on his choice:

Column Name: "Genres" //This will contain the genres of movies User genre: "Action Adventure" I want to perform cosine similarity on this data, compare user genre with genres of ...
user avatar
  • 1
1 vote
1 answer
23 views

how to find the best candidate [closed]

I am doing some work about entity disambiguation. Please suppose, there are some candidates for each entity, e.g. e1 has three candidates c1, c2, c3. Each candidate has two values: v1 and v2 We know ...
user avatar
  • 11
0 votes
1 answer
114 views

best similarity measure for images with different angles

I want to compare different images (where the images are of the same setup but the angles with which the images are taken are different). I want to obtain some sort of similarity score. I tried using ...
user avatar
  • 181
0 votes
1 answer
68 views

Non-commutative distance formula

I am trying to find a distance formula or a method that can give the non-commutative distance between two points in a feature space. Suppose there are two movies represented in an R^n feature space. ...
user avatar
1 vote
1 answer
119 views

Similarity between 2 statistical distributions

Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in ...
user avatar
  • 71
2 votes
2 answers
76 views

Find 10 most similar data points from 100?

I am trying to solve a problem where I need to group battery cells that are most similar to each other to form a reliable battery pack. It seems like a clustering problem but I only need to find the ...
user avatar
1 vote
1 answer
73 views

Why does an imbalanced data set badly effect distance measures like Mahalanobis?

I'm relatively new to data science and I am struggling to understand why the Mahalanobis distance (or any other distance measure) applied to an imbalanced data-set becomes inaccurate. I have a data ...
user avatar
  • 13
0 votes
1 answer
107 views

Siamese vs matching network for correct image category matching

I have to find the closest match between my image and bunch of already collected images of different classes in the folder. Whic meta-learning approach should I select. I am thinking about the Siamese ...
user avatar
1 vote
1 answer
298 views

Hierarchical clustering with the consensus matrix as similarity matrix

I'm following this article on consensus clustering in Python programming. On page 7 the authors state that "The consensus matrix lends itself naturally to be used as a visualization tool to help ...
user avatar
0 votes
1 answer
90 views

Similarity Matching Algorithm

I am looking for help on identifying a class of algorithm. If I have a tabular training and test set I want to know the similarity of rows based on some numeric features. The training data would be ...
user avatar
  • 326
1 vote
1 answer
44 views

How to find coherence between a large number of sentences

I have a list of sentences returned as a result of a document search algorithm. I want to determine if the results returned are semantically close/similar/coherent using some sort of metric. For a ...
user avatar
3 votes
3 answers
1k views

Dot product for similarity in word to vector computation in NLP

In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word. Uo is word vector for outside word Vc is word ...
user avatar
1 vote
1 answer
77 views

group similar subjects and train only using them

I have a dataset with 5k subjects. It's a binary classification problem where I have 3000 positive and 2000 negative subjects. Now to build a model, I don't like to train the usual way (where we build ...
user avatar
  • 2,129
1 vote
1 answer
637 views

Calculate the similarity between pairs of time series data

I have 5 pieces of time series data. It is the weekly sales of 5 different stores (A,B,C,D,E). There are no missing values. A quick visual inspection shows that these 5 pieces of time series data have ...
user avatar
  • 113
1 vote
1 answer
60 views

Scoring metric for recommendation system

I'm working on a project that involves building a news recommendation system. I've come as far as quantifying user interaction with different articles on the site into user's affinity towards atopic ...
user avatar
3 votes
1 answer
182 views

Pearson correlation with data sets that have values on different scales [closed]

I have two datasets with which I want to do a Pearson correlation analysis. I have carried out the analysis which makes sense, however I want to be sure it is valid given that both datasets have ...
user avatar
1 vote
1 answer
28 views

Similarity Threshold Standards

When using similarity measures (eg. Resnik Information Content, Cosine Similarity, etc.) for any type of data, are there any standard similarity thresholds that are used, or does it all depend on the ...
user avatar
1 vote
2 answers
284 views

Similarity between two time series with different sampling frequency, different amplitude, and different lengths but taken from the same source?

I have two files with accelerator readings and I want to get some metric/ measurement to get the similarity between these two files. I have tried Pearson’s R coefficient, dtw distance, dtw score. ...
user avatar
  • 11

1
2 3 4 5 6