Questions tagged [similarity]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
2answers
500 views

Assessing Group Similarities and Dissimilarities Post PCA

The goal is to assess similarity and dissimilarity between 6 known groups. The original data began with the 6 known groups and 2,700+ variables all on a scale of 0 to 100. I have performed PCA to ...
3
votes
4answers
6k views

How to find similarity/distance matrix with mixed Continuous and Categorical data?

Say I have a dataset like this: Hotel HasPool AvgPrice 1 1 $123 2 0 $234 3 1 $200 Currently I have broken down the ...
2
votes
1answer
89 views

How to build a symmetric similarity model on top of embeddings?

I have two equal length vectors that come out of two identical embedding layers. I want to calculate their similarity, and I don't trust the embedding layer enough to just use dot product (e.g. it's ...
1
vote
1answer
35 views

how to match a sentence to a cluster of keywords?

I have a classification problem. I have clusters called 'Experience', 'Education', 'Abilities' . The labelled data (72,000+ entries with all clusters together) with two columns looks like below. <...
0
votes
1answer
32 views

how to build word2vec content based recommendation?

I am building a content-based recommendation system for hotel accommodation. I have a hotel name, hotel description and location. I combined hotel name, description and location. Then, applied NLP and ...
0
votes
0answers
14 views

Is there a way to normalize a similarity matrix by row and column in a way such that only one entry per row or column is approximately 1

I am computing similarities between 2 vectors. My goal is to have approximately 1 matching sample with similarity ~1, for each sample, without having any samples that are similar to many other samples....
0
votes
2answers
262 views

Question about Similarity vs Dissimilarity Matrix

Right now, I'm working on a coming up with a similarity vs dissimilarity matrix for a set of data points for a clustering algorithm. My question is, if I want to use one of the many clustering ...
4
votes
2answers
51 views

Cluster elements that appear in the same lists

Suppose I have a multitude of sets with (unordered) combinations of elements and I want to determine which elements tend to appear together. For example Given the following sets: ...
4
votes
3answers
942 views

How can I group texts with similar content together?

I need to find a solution to group a corpus of texts according to document similarity. Premising I have no experience in ML - only a few readings - I'd like to know if calculating the tf-idf on each ...
2
votes
1answer
385 views

Alternate of TF-IDF

I had used TF-IDF for text similarity but the results were not so good. I tried to implement google universal encoding (tensorflow hub). The results were satisfactory but not upto the mark. Is there ...
3
votes
1answer
123 views

Comparing one small dataset with a big dataset for similar records

I create a varying small dataset (dataset: X) with 500 records in each query. Everytime I need to compare the dataset with a bigger one (dataset: A) (15 milion records) to find similar (or semi-...
1
vote
2answers
48 views

Similarity Measure of Simulated Time Series vs Observed time Series

In my work I have an observed Time Series and Simulated ones. I want to compare the Light Curves and check for similarityto find out which simulated curve fits best respectivley which parameters ...
2
votes
1answer
68 views

Similar students using Machine Learning

I have a student performance data, where I have marks of various subjects for the students and I want to find similar students with good marks in a particular subject using machine learning. How do I ...
0
votes
0answers
42 views

Semantic Search Help

There is a problem we are trying to solve where we want to do semantic search on our set of data, i.e we have a domain specific data (example: sentences talking about automobiles) Our data is just a ...
1
vote
1answer
31 views

Calculate Similarity using User's Personal Data?

I want to find out which users are similar to each other using their personal/organisational data, such as department, company, site, etc. I have this data in a boolean format, as shown below: ...
31
votes
4answers
34k views

Applications and differences for Jaccard similarity and Cosine Similarity

Jaccard similarity and cosine similarity are two very common measurements while comparing item similarities. However, I am not very clear in what situation which one should be preferable than another. ...
4
votes
4answers
2k views

Are there any good NLP APIs for comparing strings in terms of semantic similarity?

I want to create a chatbot which informs the user about traffic at the streets but not in real-time for the moment. I have created a small database with MySQL which has some data stored regarding ...
1
vote
0answers
20 views

Options to find the most similar question in a dataset of question-answer pairs?

I am building a chatbot that will only handle FAQs, but these FAQs are very specific to an organisation, so I cannot use any existing off-the-shelf solutions, or connect to question-answering APIs. I ...
15
votes
5answers
53k views

Sentence similarity prediction

I'm looking to solve the following problem: I have a set of sentences as my dataset, and I want to be able to type a new sentence, and find the sentence that the new one is the most similar to in the ...
1
vote
2answers
78 views

How can we perform STS(Semantic Textual Similarity) on UnSupervised dataset using Deep Learning?

How to implement STS(Semantic Textual Similarity) on unlabelled dataset. Dataset column contains Unique_id, text1(contains paragraph), text2(contains paragraph). Ex: Column representation: Unique_id ...
0
votes
1answer
22 views

Dissimilarity Matrix of non-metric proximity data

we currently have a coding exercise, where we are asked to implement Constant Shift Embedding (Paper). This in itself is not a big problem. For the algorithm, all you need is a symmetric non-zero ...
0
votes
1answer
23 views

looking for approaches to detecting outliers in individuals unequal sequential time series

I am looking for approaches related to outlier detection in time series. Example: A person visits hospital overtime on multiple bases and there are some measurements made (bmi, blood_pressure, ...
1
vote
2answers
54 views

How can I get an objective measure of song similarity?

I was browsing through ML project ideas and found an interesting one (just the problem statement ) which was: detecting if two songs are similar using lyrics. I found it to be an interesting idea but ...
0
votes
1answer
156 views

Assigning a new document to a cluster based on keywords extracted and tf-idf

I have about 40 clusters of documents defined by a combination of k-means clustering algorithm and hand curation. For example, some of the clusters given by k-means are too noisy so they have been ...
2
votes
1answer
40 views

Approach to semantic similarity between documents

I was wondering what approach people would take, or point me in the right direction on this challenge I have set myself. I am pretty new at this, I have covered some area but want to expand my ...
1
vote
0answers
15 views

document similarity when the document size less than 30 tokens?

I was solving a problem to compare 3 million, 2018 documents against the 2019 documents. There are three text attributes to be compared from one item against the other. I used Latent Semantic Indexing ...
0
votes
1answer
241 views

How do I calculate a similarity matrix with a Student-t kernel?

As the title says, how do I calculate a similarity matrix with an un-normalized Student-t kernel? I'm attempting to calculate Kullback-Leibler divergence for different t-SNE runs, but need a Q-matrix ...
2
votes
2answers
291 views

Similarity of words using BERTMODEL

I want to find the similarity of words using the BERT model within the NER task. I have my own dataset so, I don't want to use the pre-trained model. I do the following: ...
2
votes
1answer
33 views

Identifying documents similar to specific clusters

Through performing clustering on a set of 1,000,000 text documents, I have identified 100 clusters. I am particularly interested in, say, 10 of the clusters. Imagine, I now have an additional set of ...
0
votes
1answer
50 views

Learning similarity of representations

I am interested in a framework for learning the similarity of different input representations based on some common context. I have looked into word2vec, SVD and other recommender systems, which does ...
3
votes
1answer
2k views

Doc2vec to calculate cosine similarity - absolutely inaccurate

I'm trying to modify the Doc2vec tutorial to calculate cosine similarity and take Pandas dataframes instead of .txt documents. I ...
1
vote
1answer
44 views

Recommender system that connect users with each other , should I go for content based or collaborative filtering?

I am trying to build a system where user come on the platform and he chooses a topic(predefined few topics) and then we connect him with any random online user who chooses the same topic. Then they ...
2
votes
3answers
428 views

clustering 2-dimensional euclidean vectors - appropriate dissimilarity measure

I've got a set of approx. 50 000 2-dimensional euclidean vectors which are connected with 20 groups, i.e. each group has approx. 2500 2-dimensional euclidean vectors. My data includes endpoints ...
0
votes
0answers
13 views

How to get a percentage of similarities of new data added based on model trained using a huge training set in machine learning

I need to find similarities in newly added data using machine learning. We have a huge data set with more than 200 000 rows having following information: Name D.O.B City (From predefined list) Phone ...
0
votes
3answers
375 views

Cosine similarity vs The Levenshtein distance

Cosine similarity vs The Levenshtein distance I wanted to know what is the difference between them and in what situations they work best? As per my understanding: Cosine similarity is a measure of ...
18
votes
5answers
34k views

Best practical algorithm for sentence similarity

I have two sentences, S1 and S2, both which have a word count (usually) below 15. What are the most practically useful and successful (machine learning) algorithms, which are possibly easy to ...
0
votes
2answers
99 views

Looking for similar items in a large data set

I have a large database of people and I want to show a small number of people who are similar to each person in the database. So if one of the people was Wolfgang Mozart I would want to show Beethoven,...
2
votes
2answers
306 views

Similarity score: Can Sklearn SVR predict values greater than 1 and less than 0?

I am using svm.SVR() from scikit-learn to apply Logistic Regression on my training data to solve a similarity problem. Using GridSearchCV, I am finding the best ...
0
votes
0answers
8 views

How to draw a support set when classifying using Siamese networks without performing one shot learning?

How to perform classification on a test set with Siamese networks when I cannot afford to draw the support set from the test set itself? Possible options which come to my mind are: KNN using samples ...
5
votes
2answers
1k views

Text similarity with sentence embeddings

I'm trying to calculate similarity between texts with various lengths. My current approach is following: Using Universal Sentence Encoder, I convert text to a set of vectors. I average these vectors ...
0
votes
0answers
5 views

How to identify similar datasets based on multiple temporal variables?

I have a dataset that describes movie releases, i.e. day by day number of seats allocated to the movie and number of seats sold at different geographic locations. This dataset looks something like ...
1
vote
1answer
32 views

How to find similar points to a positive set when you don't have any negative set?

The task I'm used to do is the following. A client comes to see me with a set of clients (called positive companies) and he wants me to find other similar prospects. Usually, he also gives me a set of ...
0
votes
1answer
46 views

Clustering stores based on weekly data

I have 1 year transaction level data aggregated at a weekly level for 1000 different stores. I want to cluster similar stores based on 8 variables such as sales, customer count etc. The concern is ...
1
vote
0answers
227 views

similarity measures for categorical data in data mining

I am working on my assignment in which i have to mention 5 similarity measures for categorical and continuous data in data mining. As a beginner I tried my best and found SQUARE DISTANCE,EUCLIDEAN AND ...
0
votes
0answers
13 views

Matching resumes with jobs in a NoSQL database

I have a series of user resumes and would like to find the best matching jobs per resume, the task seems simple at first except I have a nosql db holding the jobs (1m+), my initial idea was to add a ...
-1
votes
1answer
171 views

Word Embeddings with TFIDF vectorizer

I am a beginner in machine learning. I have a large corpus of texts, divided into thematic groups. I would like to get idf values for the whole corpus, and then apply it on each group before ...
0
votes
0answers
55 views

Modifying BERT sentence encodings

I'm using BERT to encode sentences. The sentences I'm encoding are quite similar, meaning they all belong to the same overall topic. Therefor, I am using another parameter for measuring similarity. ...
2
votes
0answers
115 views

Similarity Measure Time Series

In my work I have an observed Time Series and Simulated ones. I want to compare the Light Curves and check for similarityto find out which simulated curve fits best respectivley which parameters ...
1
vote
0answers
13 views

Looking for an algorithm that compute similarity between a phrase and possible combination of tokens

I want to find similarity between a phrase and possible combination of tokens that may form the phrase. For example, phrase = 'sea surface water' Possible token = ['sea','surface land', 'surface ...
2
votes
1answer
645 views

Jaccard Similarity with Binary Data

I have 5400 rows of data and 3211 columns of attributes. The first 4 columns are ID/Name/ParentID/ObjectType - the rest of the 3207 columns are the attributes that are to be used for similarity ...