3

If you want a DL approach, I recommend substituting the tf-idf by some kind of word embeddings. For instance, you can take a pre-trained word embedding model, like glove, and average its outputs both in resume and job description, and then compute cosine similarity. However, I recommend to use a contextual word embedding (BERT-like), as the terms in resumes ...


2

Intuitively, if you normalized the vectors before using them, or if they all ended up having almost unit norm after training, then a small $l_1$ norm will imply that the angle between the vectors is small, hence the cosine similarity will be high. Conversely, almost colinear vectors will have almost equal coordinates because they all have unit length. So if ...


2

Word2vec as the name suggests will create an embedding for each word in your sentence. In order to get a sentence level embedding you would need to average (or combine in some other way) the individual embeddings together. An example of a model to generate sentence level embedding would be the Universal Sentence Encoder (USE). You may want to try it out and ...


1

It's often useful to think of simple cases, e.g. even in a 2-D (planar) case, you can't determine z. Similarity between two vectors is identical to the angle (at the origin) between them, so: for a fixed vector $b$, if $a$ had angle $x$ to $b$, then $a$ lies in one of two lines either side of $b$. if another vector $c$ has angle $y$ to $b$, then it is also ...


1

you can start by using torchscript, it may require changing ur whole code, and switching to transformers( by loading the backbone of the model and the last layers) so basically u get out from GIL interpreter, coz it does not support multithreading. by with torchscript u can run ur model in c++ env, there's also onnx which I believe it enhances performance. ...


1

There are a few models that are trained to analyse a sentence and classify each token (or recognise dependencies between words). Part of speech tagging (POS) models assign to each word its function (noun, verb, ...) - have a look at this link Dependency parsing (DP) models will recognize which words go together (in this case Angela and Merkel for instance) ...


1

So the questions asks for how to compute similarity between the organisation description and project titles. One initial thought would be to use a Doc2Vec model (concept, implementation), which will take the organisation descriptions and project titles as input and output a n-dimensional vector in semantic space for the given text. From this, you can at ...


1

There's Milvus search engine that utilizes several prominent Approximate KNN libraries such as FAISS, ANNOY and HNSW. It also handles several bookkeeping, clustering, data integrity and other tasks that you probably don't want to handle yourself. All for a performance price ofc, but if you don't want to pay it, you can always pick one of the "barebones&...


1

But the alternative is to navigate a treebank, through "type of" relationships… much, much faster and cheaper. WordNet provides exactly this: it is a lexical database in which words are grouped by synonyms, with several types of relations between groups in particular hypernyms/hyponyms (more general/more specific). The database can be downloaded ...


1

This is essentially information retrieval: usually there is a collection of documents and the goal is to find the document which is the most similar to a given query (what you call the "semantic concept"). The traditional way to do that is to convert the collection of documents as vectors, typically with TFIDF weight but there are many options (I ...


1

I don't think there's any standard, but there might be some exceptions in very specific cases where the distribution of the scores is known precisely. There's no standard because in general the optimal value of the threshold strongly depends on the task and the data. That's why thresholds are usually determined empirically based on the desired outcome. In ...


1

If you have a large corpus of text mapped with their respective topics, you can train a Siamese neural network where you have two inputs (one sentence and one topic) and output a similarity score based on whether they are related or not. This would require a good dataset with a good variety of similar and not similar pairs of (sentence, topic) to be ...


1

If you have a list of words for every topic you can indeed try to directly measure the similarity of this list against a sentence, but it's likely that a sentence doesn't always contain one of the topic words so it might not work very well. A more advanced method would be to obtain a semantic representation for every topic (or topic word) from an external ...


1

I want to know how to come up with ground truth(relevancy label) if it's not available? There's simply no way to properly evaluate a system if nobody knows what the output is supposed to be. However there are ways to work around a lack of annotated data: Ask a panel of annotators to grade the quality of the output on a sample. Disadvantage: if a relevant ...


1

They might or might not be similar, the embeddings extracted by mean pooling the BERT output usually have high cosine similarity even though the input sentences are completely different. Bert embeddings are not meant for sentence similarity task(SST), but there is some research combining Bert and SST. Here are those resources, SBERT paper: https://arxiv.org/...


1

What you are describing is one of the "standard" NLP problems faced in NLP and it usually referred to as "natural language inference" (NLI), or sometimes also as "textual entailment". There is plenty of research in this kind of task, and its variants, like cross-lingual NLI (XNLI). I suggest you have a look at nlpprogress (link) ...


Only top voted, non community-wiki answers of a minimum length are eligible