Questions tagged [similarity]
The similarity tag has no usage guidance.
246
questions
1
vote
0answers
5 views
Measuring the distance between data points based on mutual linkages
How to measure the distance between two data points (or: nodes?) based on their mutual share of linkages?
I don't know the technical term for that, so here is a fictitious example from scientific ...
2
votes
1answer
37 views
Efficiently finding most similar sports from large dataset
As part of a past interview task, I’m working with a sports streaming dataset that looks like this:
...
0
votes
0answers
20 views
Fuzzy Name String Matching
I am in need of solving the below given requirement.
Requirement:
I have two datasets which has only one column called Name. That column contains a list of user names in both the datasets so from this ...
0
votes
0answers
7 views
Calculate comparable values for semantic relations between phrases
I'm looking for the best approach to calculate a way of comparing semantic relationships between phrases. In the following extremely simplified example:
"White sharks live in oceans. All dogs ...
1
vote
0answers
10 views
How to compute similarity matrix for strings efficiently? [duplicate]
Here I'm trying to compute similarity between 1000 cross 10000 strings (using Levenshtein distance), I'm using a dataframe approach where you just need to compare n(n-1)/2 comparisons instead of n*n. ...
0
votes
1answer
17 views
applicability of relative similarity computation
I've computed the cosine similarity between a & b (=x) and ...
0
votes
0answers
9 views
Inner working behind combining two distance function as one function for similarity measure
I am comparing two images and for this I am testing various similarity function. For my case, Euclidean works much better than cosine(20% difference). However, I tried to combine two distance function ...
0
votes
0answers
15 views
Instagram Profile Similarity Features
I want to find similar IG accounts in a semantic way(not demografic like fan count, language, country,...) and thought of the following features:
Post Text Similarity (Embeddings by SBERT, averaging ...
1
vote
1answer
29 views
Why checking the distribution of data is needed before calculating Gower distance?
I read this article(Clustering datasets having both numerical and categorical variables) to learn how to perform clustering on datasets with not just numerical variables.
Before calculating the Gower ...
4
votes
1answer
42 views
encoding of text data in NLP
I'm getting data using web scraping to create a dataset. I have a 'company' column that contains the names of the companies. I would like to encode this column but i don't know how to find the ...
1
vote
0answers
37 views
Performing actual deduplication using LSH
I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
0
votes
0answers
11 views
Reducing dataset before computing similarity matrix
I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate ...
0
votes
0answers
14 views
Find minimal group of 'spanning items'
I have a group of similarity hashes which I use to detect similar items using
hamming_distance < 10.
Each time a new item is being detected by one of the item ...
1
vote
0answers
18 views
Testing similarity scores?
I need to calculate similarity between different houses using a series of attributes/properties, and to do that I need to define a certain similarity or distance function. Is there a way to test the ...
0
votes
1answer
32 views
how to calculate similarity between users based on movie ratings
Hi I am working on a movie recommendation system and I have to find alikeness between the main user and other users. For example, the main user watched 3 specific movies and rated them as 8,5,7. A ...
0
votes
0answers
31 views
Effective way to find similarity between utterance(short text) and question(long text)
The challenge I have is a bunch of questions(long text) that are closely matching with an user utterance (short text). I have tried cosine similarity & Tf-iDF, BM25,Jaccard similarity, etc., but ...
0
votes
1answer
19 views
I'm building a movie recommendation system based on genre, where the user will enter his choice and will receive recommendations based on his choice:
Column Name: "Genres" //This will contain the genres of movies
User genre: "Action Adventure"
I want to perform cosine similarity on this data, compare user genre with genres of ...
1
vote
1answer
23 views
how to find the best candidate [closed]
I am doing some work about entity disambiguation.
Please suppose, there are some candidates for each entity, e.g. e1 has three candidates c1, c2, c3.
Each candidate has two values: v1 and v2
We know ...
0
votes
0answers
15 views
Can a siamese neural network distinguish expected from unexpected changes?
Please redirect me to another stack exchange if this isn't the appropriate forum. I am interested in finding a neural network architecture that can detect distance between two inputs. As an example, ...
0
votes
1answer
58 views
best similarity measure for images with different angles
I want to compare different images (where the images are of the same setup but the angles with which the images are taken are different). I want to obtain some sort of similarity score. I tried using ...
0
votes
1answer
42 views
Non-commutative distance formula
I am trying to find a distance formula or a method that can give the non-commutative distance between two points in a feature space.
Suppose there are two movies represented in an R^n feature space. ...
1
vote
1answer
80 views
Similarity between 2 statistical distributions
Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in ...
2
votes
2answers
43 views
Find 10 most similar data points from 100?
I am trying to solve a problem where I need to group battery cells that are most similar to each other to form a reliable battery pack. It seems like a clustering problem but I only need to find the ...
0
votes
0answers
16 views
How does LDA model do inference on new documents?
Through Latent Dirichlet Allocation (LDA) one can learn topics from texts. The trained topic model can then be used on unseen data to infer to what extent each of the learned topics are represented in ...
1
vote
1answer
63 views
Why does an imbalanced data set badly effect distance measures like Mahalanobis?
I'm relatively new to data science and I am struggling to understand why the Mahalanobis distance (or any other distance measure) applied to an imbalanced data-set becomes inaccurate. I have a data ...
0
votes
1answer
55 views
Siamese vs matching network for correct image category matching
I have to find the closest match between my image and bunch of already collected images of different classes in the folder. Whic meta-learning approach should I select. I am thinking about the Siamese ...
1
vote
1answer
197 views
Hierarchical clustering with the consensus matrix as similarity matrix
I'm following this article on consensus clustering in Python programming. On page 7 the authors state that "The consensus matrix lends itself naturally to be used as a visualization tool to help ...
0
votes
0answers
45 views
How to find the most similar image between two datasets?
How do you measure the image similarity using (Mean squared error), find the most similar one, and save them as a pair of images in a folder with their similarity.
I have two images datasets:
...
0
votes
1answer
27 views
Similarity Matching Algorithm
I am looking for help on identifying a class of algorithm. If I have a tabular training and test set I want to know the similarity of rows based on some numeric features. The training data would be ...
0
votes
0answers
24 views
Is there a way to calculate the cosine distance between 2 time series?
Let's say I have time series data of City A, City B, City C & City D that looks like this:
...
1
vote
1answer
39 views
How to find coherence between a large number of sentences
I have a list of sentences returned as a result of a document search algorithm. I want to determine if the results returned are semantically close/similar/coherent using some sort of metric. For a ...
3
votes
3answers
642 views
Dot product for similarity in word to vector computation in NLP
In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word.
Uo is word vector for outside word
Vc is word ...
1
vote
1answer
40 views
group similar subjects and train only using them
I have a dataset with 5k subjects. It's a binary classification problem where I have 3000 positive and 2000 negative subjects.
Now to build a model, I don't like to train the usual way (where we build ...
1
vote
1answer
367 views
Calculate the similarity between pairs of time series data
I have 5 pieces of time series data. It is the weekly sales of 5 different stores (A,B,C,D,E). There are no missing values. A quick visual inspection shows that these 5 pieces of time series data have ...
1
vote
1answer
37 views
Scoring metric for recommendation system
I'm working on a project that involves building a news recommendation system. I've come as far as quantifying user interaction with different articles on the site into user's affinity towards atopic ...
3
votes
1answer
155 views
Pearson correlation with data sets that have values on different scales [closed]
I have two datasets with which I want to do a Pearson correlation analysis. I have carried out the analysis which makes sense, however I want to be sure it is valid given that both datasets have ...
1
vote
1answer
23 views
Similarity Threshold Standards
When using similarity measures (eg. Resnik Information Content, Cosine Similarity, etc.) for any type of data, are there any standard similarity thresholds that are used, or does it all depend on the ...
0
votes
0answers
17 views
Similarity between two sets of documents
I have 4000 tweets about a brand and 2000 comments about its products. I want to know how much the comments about the brand's products are related to the tweets it generates.
I know how to do a cosine ...
1
vote
2answers
147 views
Similarity between two time series with different sampling frequency, different amplitude, and different lengths but taken from the same source?
I have two files with accelerator readings and I want to get some metric/ measurement to get the similarity between these two files. I have tried Pearson’s R coefficient, dtw distance, dtw score. ...
1
vote
1answer
43 views
Recommender System for mostly unique user and items
I am trying to develop a recommender system for a job matching problem. My data consists of past matched candidate profiles and job profiles as well as if there was a success such that both, candidate ...
1
vote
0answers
12 views
Appearance-based hashing for similarity detection for picking the 100 most distinct images out 500 images
I would like to perform appearance-based hashing for similarity detection. I have 500 photos for each of my categories but I only want to maintain the 100 of them that are most distinct. How should I ...
2
votes
1answer
38 views
What are the Most Dissimilar MNIST Digits?
Using whatever definition of dissimilarity over sets that you'd like, what are the most dissimilar two digits in MNIST? I was thinking that a reasonable approach to answering the question would be to ...
2
votes
1answer
33 views
How is an ASR's output compared to ground truth for validation?
I am curious how it is done as I am interested in doing something similar. I have some manually transcribed data that contains tags for multiple speakers. I want to compare how well the out of the box ...
3
votes
1answer
389 views
What is the correct formula for Jaccard coefficient with integer vectors?
I understand the Jaccard index is the number of elements in common divided by the total number of distinct elements. But it seems to be some discrepancy or terminology confusion about Jaccard being ...
1
vote
1answer
29 views
Fuzzy and FuzzyWuzzy: what are the differences in text comparison?
I have found a lot of information about fuzzy logic, but less information about fuzzywuzzy. I would like to know more about this, the function which determines the logic, if possible, and understand ...
1
vote
0answers
61 views
Is the magnitude of a word vector correlated with the frequency of the word in a text?
In order to find the similarity between words, cosine similarity seems to be the most common measure to use. In a conversation that I had about this topic, someone mentioned that words that mean more ...
1
vote
0answers
15 views
Which string distance equation for fuzzy-matching person names is reliable?
A reproducible example with a small bit of R code is available in this stackoverflow post (link so I dont need to re-type out the code). The fuzzytext library in R ...
-1
votes
1answer
38 views
Equivalence of two separate data distribution
I am working in generative modelling. I have a real dataset of which I want to know the underlying distribution. For this I create a synthetic data generator which tries to mimic the real data. I ...
1
vote
2answers
164 views
State-of-the-art Python packages that can evaluate language similarity
I am trying to evaluate the likelihood of generating a specific sentence out of a large set of sentences. To do this, I start from a simple approach: training a custom n-gram language model and ...
0
votes
1answer
47 views
Modelling for similarity between two descriptions
I have a dataset of companies and research projects that they were involved in. A subset of the dataset is shown below.
I am trying to find a way to model similarity between the company and the ...
