Questions tagged [similarity]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

Negative values when calculating weighted Jaccard similarity

I have a bilateral dataset that includes countries and their the weight of their relation. I'd like to calculate the similarity of countries in 1) who their trade parterns are and 2) the weight of the ...
Johanna's user avatar
0 votes
0 answers
7 views

Collaborative Filtering Using Multiple Features

I am looking to create a recommendation system for content. The content can be likened to an instagram post (contains a caption, hashtags, an image, etc.). I want to use user-based collaborative ...
Baci's user avatar
  • 1
0 votes
0 answers
25 views

Similarity Scores between SQL tables

I'm trying to figure out the best way to get started on a project. I have two separate databases, one is a "Template" db and the other is "Content" db. For each table in the ...
Marc J's user avatar
  • 1
0 votes
0 answers
21 views

How can I measure the precision and Recall?

I did semantic search using query and the total relevant documents should be 12 documents but my model retrieve 5 relevant documents only so the irrelevant are 7 documents. how can i calculate the ...
Begnnier's user avatar
0 votes
1 answer
28 views

Synchrony vs Similarity in time series data

I would like to know what is the difference between synchrony and similarity w.r.t time series data. Upon research I get the below explanation. "Synchrony and similarity are two different ...
mohammed shoab's user avatar
0 votes
1 answer
63 views

How does RAG query affect the similarity search?

I have a RAG pipeline where I want to extract a piece of information called "X" In a regular RAG pipeline, there is a query entered by the user. Then, ...
ahmedmoh123's user avatar
1 vote
0 answers
17 views

Similarity search with text and tabular data

If I have two documents, D1 and D2 and a function f which computes the (normalized) document ...
CutePoison's user avatar
0 votes
0 answers
26 views

Interpretation of Evaluation Values of Augmented SBERT Training with EmbeddingSimilarityEvaluator()

I train a BI-Encoder to get an Augmented SBERT and I get a final training result. How can I interpret the following output of the final training result? ...
Christian01's user avatar
4 votes
1 answer
195 views

Higher level sentence similarity (meaning instead of 'just' embeddings)

I am looking for the correct model / approach for the task of checking if two sentences have the same meaning I know I can use embeddings to check similarity, but that is not what I am after. I ...
Rob Audenaerde's user avatar
0 votes
0 answers
23 views

Get accurate alias list of people without information on aliases

I'm working on a project where I have a large dataframe of paintings (from the Art500k dataset), each row corresponds with a painting, containing the author's name in the ...
me9hanics's user avatar
0 votes
0 answers
17 views

Clustering of two datasets in different years

I want to analyze two datasets by running a clustering algorithm on both and comparing the results. The two datasets have the same variables. The only difference is that one dataset is from 2010 and ...
Ahmad Bhatti's user avatar
0 votes
0 answers
4 views

Clustering or Finding Similarities Among Portfolio Allocations

I am trying to cluster a set of portfolios with percentage allocations among Stock, Bond, Other, and Cash. I am not sure what's the best way to go about this because the variables are interdependent ...
Ahmad Bhatti's user avatar
0 votes
0 answers
29 views

Proper metric for measuring the similarity between two images

I want to calculate the similarity between these two images: and These are brain topography maps and colors inside the circles represent the area being activated while watching TV. Thus I am looking ...
tail's user avatar
  • 117
0 votes
1 answer
28 views

Using text embeddings directly to compute similarity vs using them as features for a model that predicts similariy

Say you have a problem where you have a query and a set of result documents and you want to rank the result documents according to the query. Say also you have embeddings for the query and for the ...
user1893354's user avatar
0 votes
0 answers
117 views

how does netflix calculate the percentage match scores?

I know that these scores are calculated by considering the users' past interactions. But I need some details here. For example, suppose that user A has watched the movies X, Y, and Z. For a new movie ...
Sanyo Mn's user avatar
  • 123
0 votes
1 answer
90 views

What's Best way in selecting right model for document comparison

We have different pre-trained models like BERT, USE, ELMo, Word2Vec, FastText, etc.., we have documents in different sizes (large, medium, small). now, we want to do document similarity. how can we ...
tovijayak's user avatar
0 votes
0 answers
58 views

Distance Metric for a dataset with embeddings and numerical columns

I'm trying to build an Approximate Nearest Neighbours model that can fetch similar records in a dataset, that are contextually similar. For example, for a record of job=Software Engineer and age=25, a ...
Augustine Samuel's user avatar
0 votes
0 answers
26 views

Finding Look alike customers

I have a set of customers and their attributes (say spend/balance/etc.) per month. At month X something happen for some customers, but not all (call this set A). So, now I want to find customers who ...
user's user avatar
  • 2,003
5 votes
2 answers
900 views

How to handle similarity search on mixed data types vectors?

I think this question is one that many beginners run into and I could not find a decent generic guide for it. My issue is the following. I want to evaluate similarity of vectors which have mixed data ...
Chapo's user avatar
  • 53
0 votes
0 answers
813 views

Using pyspark to create a large precomputed cosine similarity matrix from text data

I would like to precompute a cosine similarity matrix for a large dataset (upwards of 5 million rows) using pyspark. Here's what I have so far. libraries: ...
cfrench's user avatar
0 votes
0 answers
88 views

Better results in Document similarity using Word2Vec

I try to cluster similar support-tickets in a technical domain. The support tickets are very domain-specific and are written in various styles, lengths, using abbreviation, etc. I made a training-...
Roland's user avatar
  • 1
1 vote
1 answer
29 views

Comparing images in N channels

I have an "image" of NxN dimensions in m channels (for reference, m is less than 17) in my training set and validation set. I would like to compare images in the training set with those in ...
Shaz's user avatar
  • 125
0 votes
1 answer
67 views

Matching items in a recommender system

I would like to ask for a proposal for a machine learning model that would be suitable for the following problem: I have a training set where each element of type A corresponds to a certain number of ...
jared's user avatar
  • 41
3 votes
1 answer
42 views

Is there a reference dataset for contextual similarity?

I'm doing some experiments with word embeddings to try to capture context-aware similarity, so that for example the word pair apple - hardware, are very dissimilar in the context of a fruit store, but ...
Jorgemar's user avatar
  • 241
0 votes
0 answers
125 views

Best string similarity metric not considering word order

I'm sorry if the title is misleading, but I didn't really know how to explain what I am searching for. I have a dataset containing two columns representing names and surnames of a bunch of people. ...
cilewu's user avatar
  • 1
0 votes
2 answers
106 views

Similarity with respect to a specific concept in text embeddings

In text embeddings, cosine similarity is often used to find texts similar to a search query. However, I don't want to find a text that is overall similar, but similar with regards to a specific ...
McLawrence's user avatar
0 votes
1 answer
39 views

Best way to compare classification output between different locations

I ran a neural network for 20+ different locations across the United States. At each location I have a list of their predictions in an array. This looks something like this... ...
Jack Cahill's user avatar
2 votes
4 answers
784 views

Algorithm to determine whether the first row in CSV is likely to be a header row or a data row

I have a fairly simple problem. I am trying to determine whether the first row in CSV is likely to be a header row or a data row. Looking at single column, the problem can be simplified to: I have a ...
David Ferris's user avatar
1 vote
3 answers
304 views

How to vectorize and speed-up double for-loop for pandas dataframe when doing text similarity scoring

I have the following dataframe: ...
illuminato's user avatar
0 votes
1 answer
71 views

What is the fastest way to detect lag and calculate cross correlation of two binary time series?

Example, arr1 = array([0,0,0,1,1,1,1,0,0,0,0,0,1,1,0,0,0,0,0,0,1,0,1,1,1,1,1,0,0]) arr2 = array([1,1,1,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0,0]) arr2 is almost perfectly correlated with ...
Imp's user avatar
  • 21
0 votes
1 answer
30 views

Find short similar sequences in long time series

Input data is a short time series, and I want to find a scalable short series that looks like the input from a long time series, should I use the sliding window, or do you have any better suggestions?
王嘉恒's user avatar
1 vote
1 answer
142 views

Nearest Neighbor Recommendation System w/ categorical variables

I would like to build a recommendation system: no ratings are available at the time of recommendation, therefore only a purely context-based recommendation system is needed as input features answers ...
alexryder's user avatar
12 votes
3 answers
3k views

Why use cosine similarity instead of scaling the vectors when calculating the similarity of vectors?

I'm watching a NLP video on Coursera. It's discussing how to calculate the similarity of two vectors. First it discusses calculating the Euclidean distance, then it discusses the cosine similarity. It ...
Allure's user avatar
  • 275
0 votes
1 answer
949 views

Adjusted Cosine Similarity With Zero Vectors

I create a recommendation engine which finds item similarities according to user ratings. I'm trying to use adjusted cosine similarity to find similarities. I follow these steps. Find mean rating of ...
Ertugrul's user avatar
0 votes
1 answer
135 views

Compute similarity with given weights for each different feature

I would like to find similar products based on the features. I have: 3 categorical features (X1,X2,X3) 1 numerical (continuous) feature (X4) 1 date feature (X5) Therefore, I want to give a pre-...
Tonino Fernandez's user avatar
1 vote
1 answer
20 views

Recommendation engine optimization

Recommendation engines typically include propensities to like the items (e.g, in a content-based approach). However, they generally do not take into account the fact that the customer may buy the ...
simon's user avatar
  • 133
2 votes
1 answer
75 views

Which method of correlation is appropriate for two paired lists of numbers?

I have a program which produces an image, and I use a metric to understand how accurate that image is. I choose five cases (A, B, C, D, E), and make a list of the accuracy metric for each case: ...
user745587's user avatar
1 vote
1 answer
60 views

A way to perform voting and select a candidate based on nearest neighbours

I'm working on a project where I use FAISS to retrieve n neighbouring vectors based on a query vector. The data in question is textual and is being embedded by using a machine learning model to create ...
mabergerx's user avatar
  • 111
0 votes
1 answer
264 views

How to measure similarities between two datasets with same features?

I have multiple datasets with the same features, a few numerical and a few categorical. The only difference is that they are market behavior for different countries. I wanted to know if there is a way ...
Sandhya Indurkar's user avatar
1 vote
1 answer
809 views

Building a graph out of a large text corpus

I'm given a large amount of documents upon which I should perform various kinds of analysis. Since the documents are to be used as a foundation of a final product, I thought about building a graph out ...
kevin_was_here's user avatar
1 vote
1 answer
114 views

What is the logic/algorithm behind 'did you mean' suggestion by search engines, command suggestion in command prompt like git?

For eg. https://stackoverflow.com/questions/307291/how-does-the-google-did-you-mean-algorithm-work this is the logic behind google's did you mean algorithm - used for spell correction suggestion. What ...
jarvis's user avatar
  • 11
1 vote
1 answer
111 views

High Performance Classification or Similarity Algorithim for Mixed Data Types?

I have a database holding 10-ish features that describe different breeds of dogs. They are mostly categorical features, but some provide ranges for values. Here's a demo representation of the database,...
CyberBully2003's user avatar
0 votes
1 answer
35 views

Word similarity considering special characteristics

I'm looking for an algorithm that computes the similarity between two strings just like the levenshtein distance. However, I want to consider the following. The <...
ahs312's user avatar
  • 1
2 votes
1 answer
2k views

Measuring similarity from massive embedded vectors

I am given a set of 10,000 journal articles, with their corresponding 100th-dimension embedded vectors. (The way they are embedded is unknown, but I'm guessing it is ...
traber's user avatar
  • 23
1 vote
1 answer
58 views

Document Similarity with User Preference

To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
JoyfulPanda's user avatar
0 votes
1 answer
346 views

How to measure the similarity between two medical images of different imaging modalities according to similar objects in both of them?

I have two series of medical images each one from different imaging modalities. According to that, I have been segmented the Region of interest (the object which appears in both modalities )using U-...
Fadil's user avatar
  • 1
0 votes
1 answer
26 views

Transformer similarity fine-tuned way too often predicts pairs as similar

I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data: ...
Simone's user avatar
  • 101
3 votes
1 answer
906 views

How to choose similarity measurement between sentences and paragraphs

Problems 1. How to find appropriate measurement method There are several ways to measure sentence similarities, but I have no idea how to find appropriate method among them for my data (sentences). ...
user avatar
0 votes
1 answer
184 views

Dynamic Time Warping (DTW) for time series with different step sizes

Is it possible to use Dynamic Time Warping (DTW) algorithm as a method to find the similarity between two time-series data that have different step/measurement sizes? For example, one is measured ...
Adel's user avatar
  • 127
0 votes
1 answer
388 views

Converting similarity value into a dissimilarity value

Suppose we have similarity values between some data point in the interval $[0, 1]$. How can I transform this similarity values into a dissimilarity values in the interval $[0, ∞]$?
Shayan's user avatar
  • 121

1
2 3 4 5 6