Questions tagged [semantic-similarity]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
31 views

Incremental semantic similarity with sentence embedding using sentence_transformers

I'm trying to find similar sentences to a given query sentence from a corpus. Also, I want to incrementally add new sentences to that corpus for future prediction without retraining the whole corpus ...
  • 1
1 vote
1 answer
118 views

Threshold determination / prediction for cosine similarity scores

Given a query sentence, we search and find similar sentences in our corpus using transformer-based models for semantic textual similarity. For one query sentence, we might get 200 similar sentences ...
0 votes
1 answer
34 views

Text cleaning when applying Sentence Similarity / Semantic Search

Do we need to apply text cleaning practices for the task of sentence similarity? Most models are being used with whole sentences that even have punctuation. Here are two example sentences that we wish ...
0 votes
0 answers
30 views

Mapping of an unseen Field/word to an existing description (in the input data), given Field and their respective descriptions as input/training data

I am working on a NLP problem. Problem Statement Given the input of fields & Labels and the respective descriptions, the goal is to the map a new unseen field to one of the most appropriate ...
  • 319
0 votes
0 answers
44 views

Parameters for training a sentence-similarity model using Bert?

I have a list of sentences : sentences = ["Missing Plate", "Plate not found"] I am trying to find the most similar sentences in the list by ...
1 vote
0 answers
38 views

Which model is better able to understand the difference that two sentences are talking about different things?

I'm currently working on the task of measuring semantic proximity between sentences. I use fasttext train _unsiupervised (skipgram) for this. I extract the sentence embeddings and then measure the ...
  • 163
0 votes
0 answers
35 views

Weighting Sentence Similarity by salience or frequency

It seems like the new standard in text search is sentence or document similarity, using things like BERT sentence embeddings. However, these don't really have a way to consider the salience of ...
1 vote
1 answer
31 views

Document Similarity with User Preference

To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
2 votes
1 answer
286 views

How to choose similarity measurement between sentences and paragraphs

Problems 1. How to find appropriate measurement method There are several ways to measure sentence similarities, but I have no idea how to find appropriate method among them for my data (sentences). ...
  • 57
0 votes
0 answers
236 views

NLP: Checking that answers to a question are correct

Question answering is a common topic within NLP, but my problem is a little different: rather than answering a question, I have a question, an (open-ended) answer, and what I want to check is if that ...
  • 101
0 votes
1 answer
30 views

Document Content

I have a set of .pdf/.docx documents with content. I need to search for the most suitable document according to a particular sentence. For instance: ...
1 vote
1 answer
18 views

Is there a way to train Doc2Vec on a corpus of docs and be able to take a novel doc and see how similar it is to the trained corpus?

I have a project idea, where I train a bunch of documents on Doc2Vec and then take a novel, input doc, and ideally be able to be told how similar it is to the docs supplied for training as a whole or ...
  • 133
1 vote
2 answers
345 views

Train a spaCy model for semantic similarity

I'm attempting to train a spaCy model for the purposes of computing semantic similarity but I'm not getting the results I would anticipate. I have created two text files that contain many sentences ...
1 vote
0 answers
25 views

Comparing the similarity structure of 2 distance matrices (computed from sentence embedding)

I apologize if this question lacks clarity, my mathematical background on the topic is limited and was hoping to find some guidance. I would like to compare 2 distance matrices that contain pair-wise ...
  • 11
0 votes
1 answer
19 views

applicability of relative similarity computation

I've computed the cosine similarity between a & b (=x) and ...
  • 285
1 vote
1 answer
78 views

BERT Optimization for Production

I'm using BERT to transform text into 768 dim vector, It's multilingual : ...
0 votes
1 answer
199 views

Cosine vs Manhattan for Text Similarity [closed]

I'm storing sentences in Elasticsearch as dense_vector field and used BERT for the embedding so each vector is 768 dim. ...
1 vote
1 answer
498 views

Cluster words into groups of similar meaning (synonyms)

How can words be clustered into groups of similar meaning (synonyms)? I started with pre-trained word embeddings (e.g., Google News), which is great, but not perfect - a limitation arises because the ...
  • 131
1 vote
1 answer
233 views

What's the best way to generate similar words?

Hi all I'm fairly up to date with all the NLP tasks out there (nlpprogress.com, paperswithcode.com) and great tools like (nltk, flair, huggingface etc). I want to take a single word, and predict a ...
  • 113
-1 votes
1 answer
36 views

How to group clusters with semantic similarity?

I have a list of job titles. I found the semantic similarity between them by using word2vec in spacy. Now I want job titles ...
  • 129
0 votes
0 answers
31 views

Return the most relevant columns for a given keyword

Suppose my database has column name and description for each column of each table. Need to design an interface where a user can enter a keyword and the interface will return the most relevance columns....
  • 101
1 vote
0 answers
20 views

Is it possible to perform Semantic Textual Similarity without using NLTK and Genism?

College restricted us to make projects in object oriented programming languages but without using any other libraries except standard ones. We can not use APIs also
1 vote
1 answer
33 views

How to determine whether a semantic concept is present in a string

I need to find a way to detect if a large string contains a specific substring. Imagine that I have a full contract page converted to string in my Python program. What I want to do is to say if a ...
1 vote
1 answer
107 views

Similarity Threshold Standards

When using similarity measures (eg. Resnik Information Content, Cosine Similarity, etc.) for any type of data, are there any standard similarity thresholds that are used, or does it all depend on the ...
1 vote
2 answers
48 views

Sentence to word similarity

Is there a way to know how much a sentence is related to a word/topic? For instance the following dataframe and the topics/attributes Romantique, ...
3 votes
1 answer
160 views

Evaluation metric for Information retrieval system

I am currently reading Semantic Product Search paper published by Amazon. They are using two evaluation subtasks matching and ranking. In matching, they tune the model hyperparameters to maximize ...
1 vote
0 answers
13 views

Appearance-based hashing for similarity detection for picking the 100 most distinct images out 500 images

I would like to perform appearance-based hashing for similarity detection. I have 500 photos for each of my categories but I only want to maintain the 100 of them that are most distinct. How should I ...
1 vote
1 answer
621 views

If i use use BERT embeddings for if cosine(sent1,sent2) > 0.9, then is it fair to assume s1 and s2 are similar

According to BERT author Jacob Devlin: I'm not sure what these vectors are, since BERT does not generate meaningful sentence vectors. It seems that this is doing average pooling over the word tokens ...
1 vote
1 answer
606 views

Should one-hot encoded categorical features needs to be scaled when used along with text feature while deriving semantic similarity?

My aim is to derive textual similarity using multiple features. Some of the features are textual for which I am using (Tfhub 2.0) Universal Sentence encoder. There are other categorical features which ...
  • 111
0 votes
1 answer
100 views

How to approach for predicting semantic similarity between two phrases

I need pointers on the latest research, tools, and techniques for predicting semantic similarity between two phrases. Problem Statement: Given two propositions A ...
0 votes
1 answer
1k views

Semantic similarity between two or more sentences

I need to determine how similar sentences (in meaning) are to one another. In order to do it, I have been considering an algorithm (cosine similarity) to determine the similarity between sentences. I ...
  • 151
0 votes
1 answer
41 views

Semantic networks: word2vec?

I have some doubts on how to represent the relationships between words in texts. Let’s suppose I have two sentences like these: ...
  • 151
0 votes
1 answer
117 views

Modelling for similarity between two descriptions

I have a dataset of companies and research projects that they were involved in. A subset of the dataset is shown below. I am trying to find a way to model similarity between the company and the ...
1 vote
0 answers
63 views

Semantic networks and conceptual graphs

I would like to use semantic networks to understand changes in texts. For example when I add/remove some words within a text. ...
  • 151
2 votes
1 answer
242 views

Semantic network using word2vec

I have thousands of headlines and I would like to build a semantic network using word2vec, specifically google news files. My sentences look like ...
  • 151
4 votes
1 answer
518 views

How to build recommendation model based on resume and job description?

How to build a model which will result in better recommendation of resumes based on the job description given? I am familiar with bow or tfidf (n-grams) approach and then take a cosine similarity but ...
  • 347
1 vote
3 answers
373 views

NLP: Compare tags semantically with machine learning? (finding synonyms)

Let's say I have multiple tags that I need to compare semantically. For example: ...
  • 111
0 votes
1 answer
128 views

How to Calculate semantic similarity between video captions?

I intend to calculate the accuracy of a caption generated by comparing it to a number of reference sentences. For example, the captions for one video are as follows: These captions are for the same ...
2 votes
1 answer
114 views

Semantic Search

There is a problem we are trying to solve where we want to do semantic search on our set of data, i.e we have a domain specific data (example: sentences talking about automobiles) Our data is just a ...
1 vote
0 answers
31 views

How we compare two paragraphs of unlabelled dataset semantically (STS)?

Column representation: Unique_id | Text1 | Text2 Unique_id 0 Text1 public show for reynolds suspension of his coaching licence. portrait sir joshua reynolds portrait of omai will get a public airing ...
3 votes
3 answers
3k views

where to store embeddings for similarity search?

I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the ...
  • 31
1 vote
1 answer
176 views

Siamese networks vs Semantic similarity (may be gensim)

I am trying to understand the Siamese networks . In this vector is calculated for an object (say an image) and a distance metric is applied (say manhatten) on two vectors produced by the neural ...