Questions tagged [similarity]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
11 votes
3 answers
1k views

Why use cosine similarity instead of scaling the vectors when calculating the similarity of vectors?

I'm watching a NLP video on Coursera. It's discussing how to calculate the similarity of two vectors. First it discusses calculating the Euclidean distance, then it discusses the cosine similarity. It ...
  • 253
0 votes
1 answer
29 views

Adjusted Cosine Similarity With Zero Vectors

I create a recommendation engine which finds item similarities according to user ratings. I'm trying to use adjusted cosine similarity to find similarities. I follow these steps. Find mean rating of ...
0 votes
0 answers
26 views

Check if two bar charts are similar

Given two bar charts, I want to check if they are "similar". By similarity I mean that they have same number of clusters with same centers. Or may be I should not use cluster terminology? In ...
  • 101
0 votes
0 answers
11 views

How do calculate the distance between the centers of these two points for a series of images, and the points between one image to another divergent?

How do I calculate the distance between the centers of these two points for a series of images, and the points between one image to another divergent?
  • 1
0 votes
0 answers
18 views

Calculating similarity score between nodes with the significance of their indexes

I have two excel files : In the first file, each row represent node number and its features as 232 1 0 0 1 0 1 477 1 1 0 0 0 0 In the second file, each ...
0 votes
1 answer
16 views

Compute similarity with given weights for each different feature

I would like to find similar products based on the features. I have: 3 categorical features (X1,X2,X3) 1 numerical (continuous) feature (X4) 1 date feature (X5) Therefore, I want to give a pre-...
0 votes
0 answers
18 views

Siamese Network constant accuracy with more training data

I am new to machine learning and I am currently trying to create a siamese network that can predict the similarity of brand logos. I have a dataset with ~210.000 brand logos. The CNN for the siamese ...
0 votes
0 answers
26 views

How To Match a User with Another User Based on Similar Spotify Music Taste?

I'm a software engineer with a background in applied math, but I'm not too familiar with Data Science so I was wondering if anyone could help me with my question. I want to match a user with another ...
  • 1
1 vote
1 answer
15 views

Recommendation engine optimization

Recommendation engines typically include propensities to like the items (e.g, in a content-based approach). However, they generally do not take into account the fact that the customer may buy the ...
  • 133
0 votes
0 answers
11 views

Metric for correlation of two paired lists of numbers

I have a program which produces an image, and I use a metric to understand how accurate that image is. I choose five cases (A, B, C, D, E), and make a list of the accuracy metric for each case: ...
1 vote
1 answer
27 views

A way to perform voting and select a candidate based on nearest neighbours

I'm working on a project where I use FAISS to retrieve n neighbouring vectors based on a query vector. The data in question is textual and is being embedded by using a machine learning model to create ...
  • 111
0 votes
1 answer
46 views

How to measure similarities between two datasets with same features?

I have multiple datasets with the same features, a few numerical and a few categorical. The only difference is that they are market behavior for different countries. I wanted to know if there is a way ...
0 votes
0 answers
6 views

Search one 2D distribution for point cluster most similar to another 2D distribution

Given a hand drawn constellation (2d distribution of points) and a map of all stars, how would you find the actual star distribution most similar to the drawn distribution? If it's helpful, suppose we ...
  • 184
0 votes
0 answers
10 views

How to cluster time series of ordered data?

There are a few hundred time series of a large set of different locations (irregularly distributed) with the following properties: ordered factor (5 levels) between 5 and 25 observations per series ...
  • 1
1 vote
1 answer
98 views

Building a graph out of a large text corpus

I'm given a large amount of documents upon which I should perform various kinds of analysis. Since the documents are to be used as a foundation of a final product, I thought about building a graph out ...
1 vote
1 answer
22 views

What is the logic/algorithm behind 'did you mean' suggestion by search engines, command suggestion in command prompt like git?

For eg. https://stackoverflow.com/questions/307291/how-does-the-google-did-you-mean-algorithm-work this is the logic behind google's did you mean algorithm - used for spell correction suggestion. What ...
  • 11
1 vote
1 answer
30 views

High Performance Classification or Similarity Algorithim for Mixed Data Types?

I have a database holding 10-ish features that describe different breeds of dogs. They are mostly categorical features, but some provide ranges for values. Here's a demo representation of the database,...
0 votes
0 answers
51 views

What ways can i find two similar sets of customers use KNN?

I have a study where i want to find users similar to a set of users (SEED). My data looks like a pivot by customer e.g. sample of SEED looks like (note i drop cust_id): ...
  • 486
0 votes
1 answer
26 views

Word similarity considering special characteristics

I'm looking for an algorithm that computes the similarity between two strings just like the levenshtein distance. However, I want to consider the following. The <...
  • 1
1 vote
1 answer
347 views

Measuring similarity from massive embedded vectors

I am given a set of 10,000 journal articles, with their corresponding 100th-dimension embedded vectors. (The way they are embedded is unknown, but I'm guessing it is ...
  • 13
0 votes
0 answers
14 views

How to generate a penalty matrix after KMeans clustering

Let say we have 9 observations that can be grouped into 3 groups using K-means clustering: ...
  • 101
1 vote
1 answer
22 views

Document Similarity with User Preference

To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
0 votes
1 answer
55 views

How to measure the similarity between two medical images of different imaging modalities according to similar objects in both of them?

I have two series of medical images each one from different imaging modalities. According to that, I have been segmented the Region of interest (the object which appears in both modalities )using U-...
  • 1
0 votes
0 answers
115 views

Comparing two images for similarity

Comparing two images for similarity What are the best softwares available for comparing two images having similarity and differences? Example: Margaret Thatcher & Enid Blyton.
0 votes
1 answer
19 views

Transformer similarity fine-tuned way too often predicts pairs as similar

I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data: ...
  • 101
0 votes
0 answers
56 views

How to find similar document (using gensim) given two or more other documents?

I am developing a similarity program to compare documents, and I’ve successfully trained my model with Gensim (TFIDF and LSI) in order to compare two documents of each other, and it works great. I can ...
  • 101
0 votes
0 answers
165 views

How to choose similarity measurement between sentences and paragraphs

Problems 1. How to find appropriate measurement method There are several ways to measure sentence similarities, but I have no idea how to find appropriate method among them for my data (sentences). ...
  • 37
0 votes
0 answers
37 views

Reproducing research results - one-shot classification for MNIST and Fashion-MNIST

I have come across two pieces of research related to one-shot classification of MNIST and Fashion-MNIST images using a 1-nearest neighbour (1-NN) classifier: [1] G. Koch, "Siamese Neural ...
  • 1
0 votes
1 answer
48 views

Dynamic Time Warping (DTW) for time series with different step sizes

Is it possible to use Dynamic Time Warping (DTW) algorithm as a method to find the similarity between two time-series data that have different step/measurement sizes? For example, one is measured ...
  • 117
0 votes
1 answer
78 views

Converting similarity value into a dissimilarity value

Suppose we have similarity values between some data point in the interval $[0, 1]$. How can I transform this similarity values into a dissimilarity values in the interval $[0, ∞]$?
  • 121
4 votes
1 answer
157 views

Has bloom filters a higher probability of collide when strings are similar

Because if it is the case, this really suits my need. I'm trying to discover popular searches on a website. For this, I'm using TopK algorithm which is based on Bloom Filter hashing. I don't want &...
  • 453
2 votes
0 answers
13 views

Measuring the distance between data points based on mutual linkages

How to measure the distance between two data points (or: nodes?) based on their mutual share of linkages? I don't know the technical term for that, so here is a fictitious example from scientific ...
  • 123
2 votes
1 answer
41 views

Efficiently finding most similar sports from large dataset

As part of a past interview task, I’m working with a sports streaming dataset that looks like this: ...
0 votes
0 answers
74 views

Fuzzy Name String Matching

I am in need of solving the below given requirement. Requirement: I have two datasets which has only one column called Name. That column contains a list of user names in both the datasets so from this ...
0 votes
0 answers
12 views

Calculate comparable values for semantic relations between phrases

I'm looking for the best approach to calculate a way of comparing semantic relationships between phrases. In the following extremely simplified example: "White sharks live in oceans. All dogs ...
1 vote
0 answers
10 views

How to compute similarity matrix for strings efficiently? [duplicate]

Here I'm trying to compute similarity between 1000 cross 10000 strings (using Levenshtein distance), I'm using a dataframe approach where you just need to compare n(n-1)/2 comparisons instead of n*n. ...
0 votes
1 answer
19 views

applicability of relative similarity computation

I've computed the cosine similarity between a & b (=x) and ...
  • 285
1 vote
1 answer
78 views

Why checking the distribution of data is needed before calculating Gower distance?

I read this article(Clustering datasets having both numerical and categorical variables) to learn how to perform clustering on datasets with not just numerical variables. Before calculating the Gower ...
4 votes
1 answer
55 views

encoding of text data in NLP

I'm getting data using web scraping to create a dataset. I have a 'company' column that contains the names of the companies. I would like to encode this column but i don't know how to find the ...
  • 43
1 vote
0 answers
116 views

Performing actual deduplication using LSH

I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
0 votes
0 answers
13 views

Reducing dataset before computing similarity matrix

I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate ...
1 vote
0 answers
19 views

Testing similarity scores?

I need to calculate similarity between different houses using a series of attributes/properties, and to do that I need to define a certain similarity or distance function. Is there a way to test the ...
0 votes
1 answer
131 views

how to calculate similarity between users based on movie ratings

Hi I am working on a movie recommendation system and I have to find alikeness between the main user and other users. For example, the main user watched 3 specific movies and rated them as 8,5,7. A ...
  • 25
0 votes
0 answers
38 views

Effective way to find similarity between utterance(short text) and question(long text)

The challenge I have is a bunch of questions(long text) that are closely matching with an user utterance (short text). I have tried cosine similarity & Tf-iDF, BM25,Jaccard similarity, etc., but ...
0 votes
1 answer
29 views

I'm building a movie recommendation system based on genre, where the user will enter his choice and will receive recommendations based on his choice:

Column Name: "Genres" //This will contain the genres of movies User genre: "Action Adventure" I want to perform cosine similarity on this data, compare user genre with genres of ...
  • 1
1 vote
1 answer
24 views

how to find the best candidate [closed]

I am doing some work about entity disambiguation. Please suppose, there are some candidates for each entity, e.g. e1 has three candidates c1, c2, c3. Each candidate has two values: v1 and v2 We know ...
  • 11
0 votes
1 answer
178 views

best similarity measure for images with different angles

I want to compare different images (where the images are of the same setup but the angles with which the images are taken are different). I want to obtain some sort of similarity score. I tried using ...
  • 181
0 votes
1 answer
87 views

Non-commutative distance formula

I am trying to find a distance formula or a method that can give the non-commutative distance between two points in a feature space. Suppose there are two movies represented in an R^n feature space. ...
1 vote
1 answer
243 views

Similarity between 2 statistical distributions

Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in ...
  • 71
2 votes
2 answers
153 views

Find 10 most similar data points from 100?

I am trying to solve a problem where I need to group battery cells that are most similar to each other to form a reliable battery pack. It seems like a clustering problem but I only need to find the ...

1
2 3 4 5 6