Questions tagged [similarity]
The similarity tag has no usage guidance.
287
questions
0
votes
0
answers
13
views
Get accurate alias list of people without information on aliases
I'm working on a project where I have a large dataframe of paintings (from the Art500k dataset), each row corresponds with a painting, containing the author's name in the ...
- 13
0
votes
0
answers
12
views
Clustering of two datasets in different years
I want to analyze two datasets by running a clustering algorithm on both and comparing the results. The two datasets have the same variables. The only difference is that one dataset is from 2010 and ...
0
votes
0
answers
4
views
Clustering or Finding Similarities Among Portfolio Allocations
I am trying to cluster a set of portfolios with percentage allocations among Stock, Bond, Other, and Cash. I am not sure what's the best way to go about this because the variables are interdependent ...
0
votes
0
answers
18
views
Proper metric for measuring the similarity between two images
I want to calculate the similarity between these two images:
and
These are brain topography maps and colors inside the circles represent the area being activated while watching TV. Thus I am looking ...
- 117
0
votes
1
answer
25
views
Using text embeddings directly to compute similarity vs using them as features for a model that predicts similariy
Say you have a problem where you have a query and a set of result documents and you want to rank the result documents according to the query. Say also you have embeddings for the query and for the ...
- 183
0
votes
0
answers
71
views
how does netflix calculate the percentage match scores?
I know that these scores are calculated by considering the users' past interactions. But I need some details here. For example, suppose that user A has watched the movies X, Y, and Z. For a new movie ...
- 123
0
votes
1
answer
62
views
What's Best way in selecting right model for document comparison
We have different pre-trained models like BERT, USE, ELMo, Word2Vec, FastText, etc..,
we have documents in different sizes (large, medium, small). now, we want to do document similarity. how can we ...
- 57
0
votes
0
answers
38
views
Distance Metric for a dataset with embeddings and numerical columns
I'm trying to build an Approximate Nearest Neighbours model that can fetch similar records in a dataset, that are contextually similar. For example, for a record of job=Software Engineer and age=25, a ...
0
votes
0
answers
17
views
Finding Look alike customers
I have a set of customers and their attributes (say spend/balance/etc.) per month.
At month X something happen for some customers, but not all (call this set A).
So, now I want to find customers who ...
- 1,973
0
votes
0
answers
28
views
Implementing efficient "Customers who viewed this, also bought that" recommendation algorithm
I'm not sure how to formulate this but I'll have a go. This code should be completely reproducible given the data below. I hope I've been clear in my question.
I'm trying to implement a recommendation ...
- 103
0
votes
0
answers
14
views
Align sub-sentences with sentences using embeddings
I would like ask for ideas
So I have sub-sentences, embedded in the context of their respective full-sentences. Then, I have other full-sentences and I would like to find
a) if they have similar sub-...
- 123
5
votes
2
answers
550
views
How to handle similarity search on mixed data types vectors?
I think this question is one that many beginners run into and I could not find a decent generic guide for it.
My issue is the following. I want to evaluate similarity of vectors which have mixed data ...
- 53
0
votes
0
answers
633
views
Using pyspark to create a large precomputed cosine similarity matrix from text data
I would like to precompute a cosine similarity matrix for a large dataset (upwards of 5 million rows) using pyspark.
Here's what I have so far.
libraries:
...
- 1
0
votes
0
answers
78
views
Better results in Document similarity using Word2Vec
I try to cluster similar support-tickets in a technical domain. The support tickets are very domain-specific and are written in various styles, lengths, using abbreviation, etc.
I made a training-...
- 1
1
vote
1
answer
27
views
Comparing images in N channels
I have an "image" of NxN dimensions in m channels (for reference, m is less than 17) in my training set and validation set. I would like to compare images in the training set with those in ...
- 125
0
votes
1
answer
50
views
Matching items in a recommender system
I would like to ask for a proposal for a machine learning model that would be suitable for the following problem:
I have a training set where each element of type A corresponds to a certain number of ...
- 41
0
votes
0
answers
16
views
How to effectively use CharBERT for text similarity?
I'm looking into CharBERT for an university project, and I noticed that it was finetuned on many tasks like sentiment analysis, NER, and so on. I tried to use it to do text similarity by using only ...
- 1
0
votes
0
answers
24
views
For Q&A NLP system, how to extract the most relevant embedding if it is a combination of top K embeddings?
From my understanding, a typical "AI" Q&A system has a (vector) database of embedded text (from a set of documents). And when a user asks a question, the user's question is embedded and ...
- 1
0
votes
0
answers
8
views
Given a Query set, find elements in a pool that are similar to the elements in the query set
I had some curiosity and I was wondering if anyone could shed some light on this. So, let's say I have a query set $Q$ that consists of the following sets of words: {...
- 21
0
votes
0
answers
22
views
getting wrong cosine similarity when using face_recognition.faceencodings
I am trying to calculate cosine similarity between two face encodings returned by face_Recognition.face encoding() which 128d vector which always is above 0.8
also is the face encoding normalized ...
3
votes
1
answer
33
views
Is there a reference dataset for contextual similarity?
I'm doing some experiments with word embeddings to try to capture context-aware similarity, so that for example the word pair apple - hardware, are very dissimilar in the context of a fruit store, but ...
- 241
0
votes
0
answers
83
views
Best string similarity metric not considering word order
I'm sorry if the title is misleading, but I didn't really know how to explain what I am searching for.
I have a dataset containing two columns representing names and surnames of a bunch of people. ...
- 1
0
votes
2
answers
92
views
Similarity with respect to a specific concept in text embeddings
In text embeddings, cosine similarity is often used to find texts similar to a search query.
However, I don't want to find a text that is overall similar, but similar with regards to a specific ...
- 101
0
votes
0
answers
21
views
Applying feedback in content based recommendation
I have a content based recommender system, which finds similar items given a list of past liked items using cosine similarity.
What would be best way to implement feedback or personalization in the ...
0
votes
1
answer
39
views
Best way to compare classification output between different locations
I ran a neural network for 20+ different locations across the United States. At each location I have a list of their predictions in an array. This looks something like this...
...
0
votes
0
answers
17
views
Find similarity in book pages
I try to find article which deals with book page similarity detection.
I already work on image similarity using embedding but in book case, I wanted to know if there are a more specific algorithm to ...
- 1
2
votes
4
answers
515
views
Algorithm to determine whether the first row in CSV is likely to be a header row or a data row
I have a fairly simple problem. I am trying to determine whether the first row in CSV is likely to be a header row or a data row. Looking at single column, the problem can be simplified to: I have a ...
- 121
0
votes
0
answers
38
views
The best approach and library for time-series similarity
I have a time-series classification problem with IoT signals. The training dataset has seven target signals.
I used tsai as a fastai/torch library, and I achieved satisfying results.
However, in a ...
- 3
0
votes
0
answers
45
views
Siamese network for a sequence-to-sequence generation
Shall I use the siamese network for a sequence-to-sequence generation problem in machine learning?
Eg:
Input 1: Sentence 1 (sequence)
Input 2: Sentence 2 (sequence)
Output: Newly Generated sentence (...
- 1
1
vote
2
answers
198
views
How to vectorize and speed-up double for-loop for pandas dataframe when doing text similarity scoring
I have the following dataframe:
...
- 306
0
votes
1
answer
49
views
What is the fastest way to detect lag and calculate cross correlation of two binary time series?
Example,
arr1 = array([0,0,0,1,1,1,1,0,0,0,0,0,1,1,0,0,0,0,0,0,1,0,1,1,1,1,1,0,0])
arr2 = array([1,1,1,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0,0])
arr2 is almost perfectly correlated with ...
- 21
0
votes
1
answer
24
views
Find short similar sequences in long time series
Input data is a short time series, and I want to find a scalable short series that looks like the input from a long time series, should I use the sliding window, or do you have any better suggestions?
- 1
1
vote
1
answer
119
views
Nearest Neighbor Recommendation System w/ categorical variables
I would like to build a recommendation system:
no ratings are available at the time of recommendation, therefore only a purely context-based recommendation system is needed
as input features answers ...
- 31
12
votes
3
answers
3k
views
Why use cosine similarity instead of scaling the vectors when calculating the similarity of vectors?
I'm watching a NLP video on Coursera. It's discussing how to calculate the similarity of two vectors. First it discusses calculating the Euclidean distance, then it discusses the cosine similarity. It ...
- 265
0
votes
1
answer
768
views
Adjusted Cosine Similarity With Zero Vectors
I create a recommendation engine which finds item similarities according to user ratings. I'm trying to use adjusted cosine similarity to find similarities. I follow these steps.
Find mean rating of ...
- 13
0
votes
1
answer
102
views
Compute similarity with given weights for each different feature
I would like to find similar products based on the features.
I have:
3 categorical features (X1,X2,X3)
1 numerical (continuous) feature (X4)
1 date feature (X5)
Therefore, I want to give a pre-...
- 101
1
vote
1
answer
20
views
Recommendation engine optimization
Recommendation engines typically include propensities to like the items (e.g, in a content-based approach). However, they generally do not take into account the fact that the customer may buy the ...
- 133
2
votes
2
answers
53
views
Which method of correlation is appropriate for two paired lists of numbers?
I have a program which produces an image, and I use a metric to understand how accurate that image is. I choose five cases (A, B, C, D, E), and make a list of the accuracy metric for each case:
...
- 21
1
vote
1
answer
47
views
A way to perform voting and select a candidate based on nearest neighbours
I'm working on a project where I use FAISS to retrieve n neighbouring vectors based on a query vector. The data in question is textual and is being embedded by using a machine learning model to create ...
- 111
0
votes
1
answer
232
views
How to measure similarities between two datasets with same features?
I have multiple datasets with the same features, a few numerical and a few categorical. The only difference is that they are market behavior for different countries. I wanted to know if there is a way ...
- 113
1
vote
1
answer
618
views
Building a graph out of a large text corpus
I'm given a large amount of documents upon which I should perform various kinds of analysis. Since the documents are to be used as a foundation of a final product, I thought about building a graph out ...
1
vote
1
answer
92
views
What is the logic/algorithm behind 'did you mean' suggestion by search engines, command suggestion in command prompt like git?
For eg. https://stackoverflow.com/questions/307291/how-does-the-google-did-you-mean-algorithm-work
this is the logic behind google's did you mean algorithm - used for spell correction suggestion.
What ...
- 11
1
vote
1
answer
97
views
High Performance Classification or Similarity Algorithim for Mixed Data Types?
I have a database holding 10-ish features that describe different breeds of dogs. They are mostly categorical features, but some provide ranges for values. Here's a demo representation of the database,...
0
votes
1
answer
35
views
Word similarity considering special characteristics
I'm looking for an algorithm that computes the similarity between two strings just like the levenshtein distance. However, I want to consider the following. The <...
- 1
2
votes
1
answer
2k
views
Measuring similarity from massive embedded vectors
I am given a set of 10,000 journal articles, with their corresponding 100th-dimension embedded vectors. (The way they are embedded is unknown, but I'm guessing it is ...
- 23
1
vote
1
answer
50
views
Document Similarity with User Preference
To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
- 111
0
votes
1
answer
279
views
How to measure the similarity between two medical images of different imaging modalities according to similar objects in both of them?
I have two series of medical images each one from different imaging modalities.
According to that, I have been segmented the Region of interest (the object which appears in both modalities )using U-...
- 1
0
votes
1
answer
25
views
Transformer similarity fine-tuned way too often predicts pairs as similar
I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data:
...
- 101
2
votes
1
answer
755
views
How to choose similarity measurement between sentences and paragraphs
Problems
1. How to find appropriate measurement method
There are several ways to measure sentence similarities, but I have no idea how to find appropriate method among them for my data (sentences).
...
user120022
0
votes
1
answer
142
views
Dynamic Time Warping (DTW) for time series with different step sizes
Is it possible to use Dynamic Time Warping (DTW) algorithm as a method to find the similarity between two time-series data that have different step/measurement sizes? For example, one is measured ...
- 117