Questions tagged [similarity]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Get accurate alias list of people without information on aliases

I'm working on a project where I have a large dataframe of paintings (from the Art500k dataset), each row corresponds with a painting, containing the author's name in the ...
me9hanics's user avatar
0 votes
0 answers
12 views

Clustering of two datasets in different years

I want to analyze two datasets by running a clustering algorithm on both and comparing the results. The two datasets have the same variables. The only difference is that one dataset is from 2010 and ...
Ahmad Bhatti's user avatar
0 votes
0 answers
4 views

Clustering or Finding Similarities Among Portfolio Allocations

I am trying to cluster a set of portfolios with percentage allocations among Stock, Bond, Other, and Cash. I am not sure what's the best way to go about this because the variables are interdependent ...
Ahmad Bhatti's user avatar
0 votes
0 answers
18 views

Proper metric for measuring the similarity between two images

I want to calculate the similarity between these two images: and These are brain topography maps and colors inside the circles represent the area being activated while watching TV. Thus I am looking ...
tail's user avatar
  • 117
0 votes
1 answer
25 views

Using text embeddings directly to compute similarity vs using them as features for a model that predicts similariy

Say you have a problem where you have a query and a set of result documents and you want to rank the result documents according to the query. Say also you have embeddings for the query and for the ...
user1893354's user avatar
0 votes
0 answers
71 views

how does netflix calculate the percentage match scores?

I know that these scores are calculated by considering the users' past interactions. But I need some details here. For example, suppose that user A has watched the movies X, Y, and Z. For a new movie ...
Sanyo Mn's user avatar
  • 123
0 votes
1 answer
62 views

What's Best way in selecting right model for document comparison

We have different pre-trained models like BERT, USE, ELMo, Word2Vec, FastText, etc.., we have documents in different sizes (large, medium, small). now, we want to do document similarity. how can we ...
tovijayak's user avatar
0 votes
0 answers
38 views

Distance Metric for a dataset with embeddings and numerical columns

I'm trying to build an Approximate Nearest Neighbours model that can fetch similar records in a dataset, that are contextually similar. For example, for a record of job=Software Engineer and age=25, a ...
Augustine Samuel's user avatar
0 votes
0 answers
17 views

Finding Look alike customers

I have a set of customers and their attributes (say spend/balance/etc.) per month. At month X something happen for some customers, but not all (call this set A). So, now I want to find customers who ...
user's user avatar
  • 1,973
0 votes
0 answers
28 views

Implementing efficient "Customers who viewed this, also bought that" recommendation algorithm

I'm not sure how to formulate this but I'll have a go. This code should be completely reproducible given the data below. I hope I've been clear in my question. I'm trying to implement a recommendation ...
Parseval's user avatar
  • 103
0 votes
0 answers
14 views

Align sub-sentences with sentences using embeddings

I would like ask for ideas So I have sub-sentences, embedded in the context of their respective full-sentences. Then, I have other full-sentences and I would like to find a) if they have similar sub-...
aqua's user avatar
  • 123
5 votes
2 answers
550 views

How to handle similarity search on mixed data types vectors?

I think this question is one that many beginners run into and I could not find a decent generic guide for it. My issue is the following. I want to evaluate similarity of vectors which have mixed data ...
Chapo's user avatar
  • 53
0 votes
0 answers
633 views

Using pyspark to create a large precomputed cosine similarity matrix from text data

I would like to precompute a cosine similarity matrix for a large dataset (upwards of 5 million rows) using pyspark. Here's what I have so far. libraries: ...
cfrench's user avatar
0 votes
0 answers
78 views

Better results in Document similarity using Word2Vec

I try to cluster similar support-tickets in a technical domain. The support tickets are very domain-specific and are written in various styles, lengths, using abbreviation, etc. I made a training-...
Roland's user avatar
  • 1
1 vote
1 answer
27 views

Comparing images in N channels

I have an "image" of NxN dimensions in m channels (for reference, m is less than 17) in my training set and validation set. I would like to compare images in the training set with those in ...
Shaz's user avatar
  • 125
0 votes
1 answer
50 views

Matching items in a recommender system

I would like to ask for a proposal for a machine learning model that would be suitable for the following problem: I have a training set where each element of type A corresponds to a certain number of ...
jared's user avatar
  • 41
0 votes
0 answers
16 views

How to effectively use CharBERT for text similarity?

I'm looking into CharBERT for an university project, and I noticed that it was finetuned on many tasks like sentiment analysis, NER, and so on. I tried to use it to do text similarity by using only ...
Gab's user avatar
  • 1
0 votes
0 answers
24 views

For Q&A NLP system, how to extract the most relevant embedding if it is a combination of top K embeddings?

From my understanding, a typical "AI" Q&A system has a (vector) database of embedded text (from a set of documents). And when a user asks a question, the user's question is embedded and ...
siddgood's user avatar
0 votes
0 answers
8 views

Given a Query set, find elements in a pool that are similar to the elements in the query set

I had some curiosity and I was wondering if anyone could shed some light on this. So, let's say I have a query set $Q$ that consists of the following sets of words: {...
M. Fire's user avatar
  • 21
0 votes
0 answers
22 views

getting wrong cosine similarity when using face_recognition.faceencodings

I am trying to calculate cosine similarity between two face encodings returned by face_Recognition.face encoding() which 128d vector which always is above 0.8 also is the face encoding normalized ...
Abhinandan Sharma's user avatar
3 votes
1 answer
33 views

Is there a reference dataset for contextual similarity?

I'm doing some experiments with word embeddings to try to capture context-aware similarity, so that for example the word pair apple - hardware, are very dissimilar in the context of a fruit store, but ...
Jorgemar's user avatar
  • 241
0 votes
0 answers
83 views

Best string similarity metric not considering word order

I'm sorry if the title is misleading, but I didn't really know how to explain what I am searching for. I have a dataset containing two columns representing names and surnames of a bunch of people. ...
cilewu's user avatar
  • 1
0 votes
2 answers
92 views

Similarity with respect to a specific concept in text embeddings

In text embeddings, cosine similarity is often used to find texts similar to a search query. However, I don't want to find a text that is overall similar, but similar with regards to a specific ...
McLawrence's user avatar
0 votes
0 answers
21 views

Applying feedback in content based recommendation

I have a content based recommender system, which finds similar items given a list of past liked items using cosine similarity. What would be best way to implement feedback or personalization in the ...
Kei Shuri's user avatar
0 votes
1 answer
39 views

Best way to compare classification output between different locations

I ran a neural network for 20+ different locations across the United States. At each location I have a list of their predictions in an array. This looks something like this... ...
Jack Cahill's user avatar
0 votes
0 answers
17 views

Find similarity in book pages

I try to find article which deals with book page similarity detection. I already work on image similarity using embedding but in book case, I wanted to know if there are a more specific algorithm to ...
dev dev's user avatar
2 votes
4 answers
515 views

Algorithm to determine whether the first row in CSV is likely to be a header row or a data row

I have a fairly simple problem. I am trying to determine whether the first row in CSV is likely to be a header row or a data row. Looking at single column, the problem can be simplified to: I have a ...
David Ferris's user avatar
0 votes
0 answers
38 views

The best approach and library for time-series similarity

I have a time-series classification problem with IoT signals. The training dataset has seven target signals. I used tsai as a fastai/torch library, and I achieved satisfying results. However, in a ...
AbelAI's user avatar
  • 3
0 votes
0 answers
45 views

Siamese network for a sequence-to-sequence generation

Shall I use the siamese network for a sequence-to-sequence generation problem in machine learning? Eg: Input 1: Sentence 1 (sequence) Input 2: Sentence 2 (sequence) Output: Newly Generated sentence (...
Pradeep's user avatar
1 vote
2 answers
198 views

How to vectorize and speed-up double for-loop for pandas dataframe when doing text similarity scoring

I have the following dataframe: ...
illuminato's user avatar
0 votes
1 answer
49 views

What is the fastest way to detect lag and calculate cross correlation of two binary time series?

Example, arr1 = array([0,0,0,1,1,1,1,0,0,0,0,0,1,1,0,0,0,0,0,0,1,0,1,1,1,1,1,0,0]) arr2 = array([1,1,1,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0,0]) arr2 is almost perfectly correlated with ...
Imp's user avatar
  • 21
0 votes
1 answer
24 views

Find short similar sequences in long time series

Input data is a short time series, and I want to find a scalable short series that looks like the input from a long time series, should I use the sliding window, or do you have any better suggestions?
王嘉恒's user avatar
1 vote
1 answer
119 views

Nearest Neighbor Recommendation System w/ categorical variables

I would like to build a recommendation system: no ratings are available at the time of recommendation, therefore only a purely context-based recommendation system is needed as input features answers ...
alexryder's user avatar
12 votes
3 answers
3k views

Why use cosine similarity instead of scaling the vectors when calculating the similarity of vectors?

I'm watching a NLP video on Coursera. It's discussing how to calculate the similarity of two vectors. First it discusses calculating the Euclidean distance, then it discusses the cosine similarity. It ...
Allure's user avatar
  • 265
0 votes
1 answer
768 views

Adjusted Cosine Similarity With Zero Vectors

I create a recommendation engine which finds item similarities according to user ratings. I'm trying to use adjusted cosine similarity to find similarities. I follow these steps. Find mean rating of ...
Ertugrul's user avatar
0 votes
1 answer
102 views

Compute similarity with given weights for each different feature

I would like to find similar products based on the features. I have: 3 categorical features (X1,X2,X3) 1 numerical (continuous) feature (X4) 1 date feature (X5) Therefore, I want to give a pre-...
Tonino Fernandez's user avatar
1 vote
1 answer
20 views

Recommendation engine optimization

Recommendation engines typically include propensities to like the items (e.g, in a content-based approach). However, they generally do not take into account the fact that the customer may buy the ...
simon's user avatar
  • 133
2 votes
2 answers
53 views

Which method of correlation is appropriate for two paired lists of numbers?

I have a program which produces an image, and I use a metric to understand how accurate that image is. I choose five cases (A, B, C, D, E), and make a list of the accuracy metric for each case: ...
user745587's user avatar
1 vote
1 answer
47 views

A way to perform voting and select a candidate based on nearest neighbours

I'm working on a project where I use FAISS to retrieve n neighbouring vectors based on a query vector. The data in question is textual and is being embedded by using a machine learning model to create ...
mabergerx's user avatar
  • 111
0 votes
1 answer
232 views

How to measure similarities between two datasets with same features?

I have multiple datasets with the same features, a few numerical and a few categorical. The only difference is that they are market behavior for different countries. I wanted to know if there is a way ...
Sandhya Indurkar's user avatar
1 vote
1 answer
618 views

Building a graph out of a large text corpus

I'm given a large amount of documents upon which I should perform various kinds of analysis. Since the documents are to be used as a foundation of a final product, I thought about building a graph out ...
kevin_was_here's user avatar
1 vote
1 answer
92 views

What is the logic/algorithm behind 'did you mean' suggestion by search engines, command suggestion in command prompt like git?

For eg. https://stackoverflow.com/questions/307291/how-does-the-google-did-you-mean-algorithm-work this is the logic behind google's did you mean algorithm - used for spell correction suggestion. What ...
jarvis's user avatar
  • 11
1 vote
1 answer
97 views

High Performance Classification or Similarity Algorithim for Mixed Data Types?

I have a database holding 10-ish features that describe different breeds of dogs. They are mostly categorical features, but some provide ranges for values. Here's a demo representation of the database,...
CyberBully2003's user avatar
0 votes
1 answer
35 views

Word similarity considering special characteristics

I'm looking for an algorithm that computes the similarity between two strings just like the levenshtein distance. However, I want to consider the following. The <...
ahs312's user avatar
  • 1
2 votes
1 answer
2k views

Measuring similarity from massive embedded vectors

I am given a set of 10,000 journal articles, with their corresponding 100th-dimension embedded vectors. (The way they are embedded is unknown, but I'm guessing it is ...
traber's user avatar
  • 23
1 vote
1 answer
50 views

Document Similarity with User Preference

To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
JoyfulPanda's user avatar
0 votes
1 answer
279 views

How to measure the similarity between two medical images of different imaging modalities according to similar objects in both of them?

I have two series of medical images each one from different imaging modalities. According to that, I have been segmented the Region of interest (the object which appears in both modalities )using U-...
Fadil's user avatar
  • 1
0 votes
1 answer
25 views

Transformer similarity fine-tuned way too often predicts pairs as similar

I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data: ...
Simone's user avatar
  • 101
2 votes
1 answer
755 views

How to choose similarity measurement between sentences and paragraphs

Problems 1. How to find appropriate measurement method There are several ways to measure sentence similarities, but I have no idea how to find appropriate method among them for my data (sentences). ...
user avatar
0 votes
1 answer
142 views

Dynamic Time Warping (DTW) for time series with different step sizes

Is it possible to use Dynamic Time Warping (DTW) algorithm as a method to find the similarity between two time-series data that have different step/measurement sizes? For example, one is measured ...
Adel's user avatar
  • 117

1
2 3 4 5 6