Questions tagged [similarity]
The similarity tag has no usage guidance.
1
vote
0answers
12 views
How to check the similarity between two transition matrixes
I have two transition matrixes in which the probabilities for the transition between each from state to each to states are ...
1
vote
0answers
10 views
How can I select a similarity threshold value for strings?
I have several large data sets of names (i.e., individuals, groups, and companies) where there is a lot of similar, but not quite the same data. For example, a corporation might be listed as "Acme ...
1
vote
0answers
16 views
patient grouping and counting metrics spark
I am working on a project to define if kids have a given disease. The idea is to identify if the kids are sick and group them by disease. I have a large number of patients (kids) and I calculated ...
-1
votes
1answer
32 views
Handling Categorical Data and Numerical Data Together [on hold]
I have a Dataframe containing 3 features: Product_detail, S.I_Units and Value.
...
0
votes
1answer
37 views
Nearest Neighbors on mixed data types in high dimensions
I would like to be able to use nearest neighbors to attempt to find the most similar samples to a subclass of samples (think treated vs untreated) in a dataset with continuous, categorical, and text ...
0
votes
2answers
28 views
Use text similarity (cosine) instead of machine learning to classify companies into industries
I am building an industry classifier. I.e. classifying companies into industries based on a company's description. Each company can only have one industry.
I took 2000 companies and assigned them ...
1
vote
1answer
18 views
How to find similarity of two series over time containing periodic trends?
Considering the data is received from a streaming source each second.How to distinguish if both the line graphs 'look' same/different in real time, statically, like the picture given below
Edit:
1....
0
votes
0answers
29 views
Fast RBF kernel calculations in python?
I would like to determine the similarity (Gram) matrix of dimensions 20.000 x 20.000 with a RBF dot product function with 4.800 feature vectors dimension. It takes too much (more than 1 day) on my PC ...
0
votes
0answers
9 views
TNIC scores: why would they choose only 25% of unique words
The paper described here is Gerard Hoberg and Gordon Phillips: Text-Based Network Industries and Endogenous Product Differentiation. My question is about the way they construct the bag of words for ...
1
vote
0answers
12 views
Best way to find dissimilarity in a 6x2 DataFrame?
I'm new to data science and am currently learning different techniques that I can do with Python. Currently, I'm trying it out with Spotify's API for my own playlists.
The goal is to find the most ...
0
votes
0answers
3 views
Efficient Similarity Indexing and Searching in High Dimensions
I found this recent document, containing simple algorithm that properly indexes high dimensional vectors for binary search later.
Algorithm basically uses random partition forest that randomly ...
0
votes
0answers
20 views
Evaluation of sentence clustering using pair-wise similarities
I am trying to evaluate different approaches for (text) sentence clustering. I have a dataset with (incomplete) pair-wise similarities between sentences, e.g.
sentence1 | sentence2 | similarity-score ...
0
votes
0answers
12 views
min-hash n-gram letter vs word shingling
I am working on a document similarity project. I am trying to use min-hash to generate similarity comparison between many docs. Rather then using 2-gram words, I wanted to use 2-gram letter pairs as ...
0
votes
0answers
15 views
Possible metrics to judge closeness of predicted curve/points compared to source curve/points?
I have a question. Pardon me if I make any mistakes in posing the question, for I am a beginner in this field.
I have a time series dataset of this form :
Example :
...
1
vote
1answer
18 views
Identifying documents similar to specific clusters
Through performing clustering on a set of 1,000,000 text documents, I have identified 100 clusters. I am particularly interested in, say, 10 of the clusters. Imagine, I now have an additional set of ...
1
vote
1answer
28 views
Recommender system that connect users with each other , should I go for content based or collaborative filtering?
I am trying to build a system where user come on the platform and he chooses a topic(predefined few topics) and then we connect him with any random online user who chooses the same topic. Then they ...
0
votes
1answer
102 views
Similarity of Dictionaries Using Cosine Similarity
I'm trying to use Cosine Similarity in Python to compare users, but can't seem to get it quite right.
...
0
votes
0answers
10 views
Clustering images of same group
I have 3 groups of images (different domains)
like let's say for example (1K images in night, 1K images in day, 1K images in rainy weather).
I am looking for a way to do clustering for these 3 groups,...
1
vote
1answer
49 views
Adaboost vs Gradient Boosting
How AdaBoost is different than Gradient Boosting algorithm since both of them works on Boosting technique?
I could not figure out actual difference between these both algorithms from theory point of ...
1
vote
1answer
43 views
Creating similarity metric with Doc2Vec and additional features
I have a dataset which contains many features. Each record is company that has many features.
For example...
Company A:
Keywords - data, big data, tableau, dashboards, etc.
Industry - Information ...
1
vote
0answers
17 views
Can I sum up feature vectors of a user‘s collection?
I want to find items that are similar to items users already have in their collection. Every item has attributes, so I created feature vectors where every element of the vector represents an attribute ...
1
vote
0answers
14 views
Efficient search for a Triples data [closed]
I have a text file which consists of ~10k triples in the format,
Let say I need to extract all the relevant triples for the query "Tom and Jerry likes to play football", with this I am extracting NP(...
2
votes
0answers
108 views
Alternate of TF-IDF
I had used TF-IDF for text similarity but the results were not so good. I tried to implement google universal encoding (tensorflow hub). The results were satisfactory but not upto the mark.
Is there ...
1
vote
1answer
23 views
Significance of log in DCG metric?
I am trying to make logical sense about why DCG is metric is formulated as such but I am not able to understand what's the need of taking the log of (1 + rank) term....
1
vote
0answers
47 views
Counting similarity between two GIT commits
I am writing as part of my master thesis work an analysis of developers' work. I am looking for a new parameters to be consumed in Topological Data Analysis process. I would like to compare a ...
1
vote
1answer
21 views
Appropriate similarity measure that highlights symmetrical values of features
I am trying to find an appropriate distance measure that reflects the differences of the vectors seen in the image below:
The green vector is compared with the blue one and the orange one.
Most of ...
0
votes
0answers
38 views
Implementing triplet loss for an arbitrarily large number of class
I'm trying to implement a triplet loss for a large number of class. I've checked out a few of its implementations in PyTorch. However, they are usually used for datasets with a limited number of ...
2
votes
0answers
57 views
Siamese Network Triplet Loss for unlabeled images
Hi I am rather new to machine learning so I'm not really sure whether it's possible or not. Right now I'm trying to train a siamese network to identify whether an image is the same or not(by one-shot ...
0
votes
1answer
218 views
Text Similarities: which nlp methods to use?
I have data where there is text for each user A visiting a business B. I want to find similarity between each user using their text.
Question 1: Which NLP method should I start with?
I have tried ...
1
vote
2answers
39 views
Finding best cluster after running K-Means clustering
I have bunch of text which I want to segregate based on semantic similarity. Running through K-Means, I was able to divide the complete text into different clusters.
However, I still need to find ...
0
votes
1answer
296 views
Can cosine similarity be applied to multidimensional matrices?
I'm trying to find the similarity between two 4D matrices. Because cosine similarity takes the dot product of the input matrices, the result is inevitably a matrix. Is there a way to get a scalar ...
1
vote
1answer
27 views
Clustering 1-gram Strings
I have a big list of mail addresses (around one million) and I want to use/adapt/create an algorithm to find similarities between them (basically to cluster them).
All the algorithms I've checked so ...
2
votes
1answer
50 views
Finding the most phonetically similar word from WordNet
Besides soundex and other libraries, that take two words and determine whether they are similar, is there any way to find the most similar sounding word from WordNet, for a given word? I tried to use ...
2
votes
3answers
107 views
clustering 2-dimensional euclidean vectors - appropriate dissimilarity measure
I've got a set of approx. 50 000 2-dimensional euclidean vectors which are connected with 20 groups, i.e. each group has approx. 2500 2-dimensional euclidean vectors. My data includes endpoints ...
0
votes
0answers
27 views
Estimate the number of very distinct elements inside a dataset
Lets take for example the MNIST dataset, this dataset is composed of 60 000 training samples each sample is a handwritten digit between 0 and 9. however, we may find several samples inside this ...
0
votes
1answer
18 views
Voting patterns similarities
I'm interested in any research materials on voting patterns.
I have a data set of how PMs (members of parliament) voted in my country during last couple of years. Each PM has 3 buttons: Yes, No, ...
1
vote
2answers
31 views
Grouping company information
I have 3 different datasets with company information, in all of them I have company name, but is not perfect: For example:
Dataset A: Company name: Facebook
Dataset B: Company name: Facebook, Inc
...
3
votes
1answer
95 views
Are there any very good APIs for matching similar images?
I am working with Python. I have 3000 thousands images of front-faced watches like the following ones: Watch1, Watch2, Watch3.
I want to find an API which receives this collection of photos or even ...
2
votes
1answer
111 views
How can I measure the similarity between 2 IP addresses? Is there any code to re-use?
I need to measure the similarity of 2 IP addresses. I could not find any sample code in scala or other languages to find the distance between 2 IPs.
3
votes
2answers
364 views
Are there any good NLP APIs for comparing strings in terms of semantic similarity?
I want to create a chatbot which informs the user about traffic at the streets but not in real-time for the moment. I have created a small database with MySQL which has some data stored regarding ...
0
votes
1answer
59 views
Image Matching to solve captcha
I am building a bot with python and I need some system to solve captchas like these:
I think I need a deep learning algorithm, but coding one is a pain in the ass. Is there any easy solution to this?
...
1
vote
2answers
142 views
intra-cluster similarity metric
I have some observations belonging to groups, and I want to compute the similarity of those observations within the different groups in order to tell that observations within specific group have ...
4
votes
2answers
1k views
Unsupervised Anomaly Detection in Images
I would like to detect defects/anomalies in images. Due to the lack of images with anomalies, I try to solve the problem in an unsupervised manner. Until now, I trained a variational autoencoder ...
1
vote
1answer
412 views
How to find similar time series?
I've got a collection of yearly data (one value per year per category), and I'd like to find series that are most similar to one another.
Example data is here.
I don't know much about data science, ...
0
votes
0answers
82 views
User-user suggestions with collaborative filtering
I have a binary matrix N x N where both rows and columns represent users of a website. If matix[i,j] = 1 it means that ...
1
vote
0answers
315 views
Deep Learning - Find most similar images - Triplets vs Pairs
I am working with Python, scikit-learn, keras and with 450x540 rgb images of front-faced watches (e.g. Watch_1, Watch_2).
My aim to run an autoencoder or a Siemese Neural Network to find the most ...
0
votes
4answers
173 views
Similarity metric for clustering
I am working on a clustering problem. I am not able to find the right similarity metric for my system.
I have n nodes with ordered vector (eg: [1,0,0,0,1,0,.....] "1" represents represent presence of ...
3
votes
2answers
68 views
Collection Of Variable Length Sequences and Descriptions: A Search Problem
I have a tough problem and need some advice:
Suppose I have a collection of variable length sequences, many of which are unique -- imagine the moves to a chess game, eg
d4 Nf6
c4 g6
Nc3 Bg7
...
2
votes
1answer
310 views
Which dissimilarity/similarity measure use after a dimension reduction ( PCA / AutoEncoder / … )?
Each problem required its own similarity/dissimilarity measure. Imagine we are dealing with dataset composed with vector of real. I suppose that we mostly use the euclidean distance especially in low ...
1
vote
2answers
72 views
How to validate different method of clustering
I have applied K-means and Hierarchical Agglomerative clustering method on some data and clustered them into 5 groups. For validation (agreement) purpose I used this formula:
...
