Questions tagged [similarity]
The similarity tag has no usage guidance.
242
questions
0
votes
0answers
12 views
Which algorithm should I use to ensure a (near-) optimum order scheduling for a packaging line?
I'm looking for an algorithm or methodology to solve order scheduling on a packaging line, but I'm struggling to find the right direction/search terms.
The simplified version of the problem is as ...
1
vote
1answer
18 views
Why checking the distribution of data is needed before calculating Gower distance?
I read this article(Clustering datasets having both numerical and categorical variables) to learn how to perform clustering on datasets with not just numerical variables.
Before calculating the Gower ...
4
votes
1answer
41 views
encoding of text data in NLP
I'm getting data using web scraping to create a dataset. I have a 'company' column that contains the names of the companies. I would like to encode this column but i don't know how to find the ...
1
vote
0answers
11 views
Performing actual deduplication using LSH
I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
0
votes
0answers
11 views
Reducing dataset before computing similarity matrix
I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate ...
0
votes
0answers
14 views
Find minimal group of 'spanning items'
I have a group of similarity hashes which I use to detect similar items using
hamming_distance < 10.
Each time a new item is being detected by one of the item ...
1
vote
0answers
18 views
Testing similarity scores?
I need to calculate similarity between different houses using a series of attributes/properties, and to do that I need to define a certain similarity or distance function. Is there a way to test the ...
0
votes
1answer
30 views
how to calculate similarity between users based on movie ratings
Hi I am working on a movie recommendation system and I have to find alikeness between the main user and other users. For example, the main user watched 3 specific movies and rated them as 8,5,7. A ...
0
votes
0answers
24 views
Effective way to find similarity between utterance(short text) and question(long text)
The challenge I have is a bunch of questions(long text) that are closely matching with an user utterance (short text). I have tried cosine similarity & Tf-iDF, BM25,Jaccard similarity, etc., but ...
0
votes
1answer
19 views
I'm building a movie recommendation system based on genre, where the user will enter his choice and will receive recommendations based on his choice:
Column Name: "Genres" //This will contain the genres of movies
User genre: "Action Adventure"
I want to perform cosine similarity on this data, compare user genre with genres of ...
1
vote
1answer
23 views
how to find the best candidate [closed]
I am doing some work about entity disambiguation.
Please suppose, there are some candidates for each entity, e.g. e1 has three candidates c1, c2, c3.
Each candidate has two values: v1 and v2
We know ...
0
votes
0answers
14 views
Can a siamese neural network distinguish expected from unexpected changes?
Please redirect me to another stack exchange if this isn't the appropriate forum. I am interested in finding a neural network architecture that can detect distance between two inputs. As an example, ...
0
votes
1answer
36 views
best similarity measure for images with different angles
I want to compare different images (where the images are of the same setup but the angles with which the images are taken are different). I want to obtain some sort of similarity score. I tried using ...
0
votes
1answer
27 views
Non-commutative distance formula
I am trying to find a distance formula or a method that can give the non-commutative distance between two points in a feature space.
Suppose there are two movies represented in an R^n feature space. ...
1
vote
1answer
68 views
Similarity between 2 statistical distributions
Is there any index that measures similarity between 2 gaussian distributions of 1-D data (may have slightly different number of points) considering their mean shift, variance shift, difference in ...
2
votes
2answers
37 views
Find 10 most similar data points from 100?
I am trying to solve a problem where I need to group battery cells that are most similar to each other to form a reliable battery pack. It seems like a clustering problem but I only need to find the ...
0
votes
0answers
15 views
How does LDA model do inference on new documents?
Through Latent Dirichlet Allocation (LDA) one can learn topics from texts. The trained topic model can then be used on unseen data to infer to what extent each of the learned topics are represented in ...
1
vote
1answer
56 views
Why does an imbalanced data set badly effect distance measures like Mahalanobis?
I'm relatively new to data science and I am struggling to understand why the Mahalanobis distance (or any other distance measure) applied to an imbalanced data-set becomes inaccurate. I have a data ...
0
votes
1answer
34 views
Siamese vs matching network for correct image category matching
I have to find the closest match between my image and bunch of already collected images of different classes in the folder. Whic meta-learning approach should I select. I am thinking about the Siamese ...
1
vote
1answer
137 views
Hierarchical clustering with the consensus matrix as similarity matrix
I'm following this article on consensus clustering in Python programming. On page 7 the authors state that "The consensus matrix lends itself naturally to be used as a visualization tool to help ...
0
votes
0answers
28 views
How to find the most similar image between two datasets?
How do you measure the image similarity using (Mean squared error), find the most similar one, and save them as a pair of images in a folder with their similarity.
I have two images datasets:
...
0
votes
1answer
18 views
Similarity Matching Algorithm
I am looking for help on identifying a class of algorithm. If I have a tabular training and test set I want to know the similarity of rows based on some numeric features. The training data would be ...
0
votes
0answers
20 views
Is there a way to calculate the cosine distance between 2 time series?
Let's say I have time series data of City A, City B, City C & City D that looks like this:
...
1
vote
1answer
31 views
How to find coherence between a large number of sentences
I have a list of sentences returned as a result of a document search algorithm. I want to determine if the results returned are semantically close/similar/coherent using some sort of metric. For a ...
3
votes
3answers
347 views
Dot product for similarity in word to vector computation in NLP
In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word.
Uo is word vector for outside word
Vc is word ...
1
vote
1answer
38 views
group similar subjects and train only using them
I have a dataset with 5k subjects. It's a binary classification problem where I have 3000 positive and 2000 negative subjects.
Now to build a model, I don't like to train the usual way (where we build ...
1
vote
1answer
160 views
Calculate the similarity between pairs of time series data
I have 5 pieces of time series data. It is the weekly sales of 5 different stores (A,B,C,D,E). There are no missing values. A quick visual inspection shows that these 5 pieces of time series data have ...
1
vote
1answer
29 views
Scoring metric for recommendation system
I'm working on a project that involves building a news recommendation system. I've come as far as quantifying user interaction with different articles on the site into user's affinity towards atopic ...
3
votes
1answer
137 views
Pearson correlation with data sets that have values on different scales [closed]
I have two datasets with which I want to do a Pearson correlation analysis. I have carried out the analysis which makes sense, however I want to be sure it is valid given that both datasets have ...
1
vote
1answer
19 views
Similarity Threshold Standards
When using similarity measures (eg. Resnik Information Content, Cosine Similarity, etc.) for any type of data, are there any standard similarity thresholds that are used, or does it all depend on the ...
0
votes
0answers
16 views
Similarity between two sets of documents
I have 4000 tweets about a brand and 2000 comments about its products. I want to know how much the comments about the brand's products are related to the tweets it generates.
I know how to do a cosine ...
0
votes
1answer
55 views
Similarity between two time series with different sampling frequency, different amplitude, and different lengths but taken from the same source?
I have two files with accelerator readings and I want to get some metric/ measurement to get the similarity between these two files. I have tried Pearson’s R coefficient, dtw distance, dtw score. ...
1
vote
1answer
39 views
Recommender System for mostly unique user and items
I am trying to develop a recommender system for a job matching problem. My data consists of past matched candidate profiles and job profiles as well as if there was a success such that both, candidate ...
0
votes
0answers
9 views
Appearance-based hashing for similarity detection for picking the 100 most distinct images out 500 images
I would like to perform appearance-based hashing for similarity detection. I have 500 photos for each of my categories but I only want to maintain the 100 of them that are most distinct. How should I ...
2
votes
1answer
34 views
What are the Most Dissimilar MNIST Digits?
Using whatever definition of dissimilarity over sets that you'd like, what are the most dissimilar two digits in MNIST? I was thinking that a reasonable approach to answering the question would be to ...
2
votes
1answer
30 views
How is an ASR's output compared to ground truth for validation?
I am curious how it is done as I am interested in doing something similar. I have some manually transcribed data that contains tags for multiple speakers. I want to compare how well the out of the box ...
3
votes
1answer
265 views
What is the correct formula for Jaccard coefficient with integer vectors?
I understand the Jaccard index is the number of elements in common divided by the total number of distinct elements. But it seems to be some discrepancy or terminology confusion about Jaccard being ...
0
votes
0answers
21 views
product recommendation of a single product based on customer similarity?
I have been wondering how you can build a model to recommend only one single product to a bunch of customers. So basically the question that I would like to answer with this model is to have a ranking ...
1
vote
1answer
23 views
Fuzzy and FuzzyWuzzy: what are the differences in text comparison?
I have found a lot of information about fuzzy logic, but less information about fuzzywuzzy. I would like to know more about this, the function which determines the logic, if possible, and understand ...
1
vote
0answers
40 views
Is the magnitude of a word vector correlated with the frequency of the word in a text?
In order to find the similarity between words, cosine similarity seems to be the most common measure to use. In a conversation that I had about this topic, someone mentioned that words that mean more ...
1
vote
0answers
12 views
Which string distance equation for fuzzy-matching person names is reliable?
A reproducible example with a small bit of R code is available in this stackoverflow post (link so I dont need to re-type out the code). The fuzzytext library in R ...
0
votes
0answers
9 views
Link Recommendation System
I am trying to build a Recommendation system on my website for recommending similar articles to the user. Eg: Lets say a user is reading an article about sports on a news website. The next article ...
-1
votes
1answer
37 views
Equivalence of two separate data distribution
I am working in generative modelling. I have a real dataset of which I want to know the underlying distribution. For this I create a synthetic data generator which tries to mimic the real data. I ...
1
vote
2answers
121 views
State-of-the-art Python packages that can evaluate language similarity
I am trying to evaluate the likelihood of generating a specific sentence out of a large set of sentences. To do this, I start from a simple approach: training a custom n-gram language model and ...
0
votes
1answer
30 views
Modelling for similarity between two descriptions
I have a dataset of companies and research projects that they were involved in. A subset of the dataset is shown below.
I am trying to find a way to model similarity between the company and the ...
0
votes
0answers
58 views
Why do all DTW-packages use different step-patterns and distance-methods?
I already posted it on StackOverflow. I was not sure if this was the right place.
I have several questions about the algorithm DTW. I tried different Python-Packages
DTAI-Distance
DTW-Python
First ...
3
votes
2answers
2k views
Similarity Measure between two feature vectors
I have face identification system with following details:
VGG16 model for feature extraction
512 dimensional feature vector (...
0
votes
1answer
26 views
How to calculate similarity between 2 users based on the images they share?
Say there are 2 users, A and B, and they each shared 10 images (in some social media site), which I have collected in 2 folders separately.
I want to calculate the similarity between the 2 users based ...
1
vote
0answers
29 views
Pairwise 3D object correlation between 2 objects
I have a dataset which contains 3D CT scans from different patients along with the segmenation masks of a certain organ. The 3D scans have been drawn each day for a period of 30 days for each patient. ...
1
vote
1answer
223 views
How to figure out if two sentences have the same meaning with AI?
I have two sentences which might be similar in meaning.
Are there a useful and successful (machine learning) algorithms, which is able to determine the semantic similarity?
Are there any approaches ...
