4

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector. The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to ...


4

A way to speed up this process is to preprocess the large dataset, the goal being to store the documents from A in a way which avoids a lot of useless comparisons. Store each document from A in an inverted index $m$, so that for any word $w$ $m[w]$ is the list of all documents in A which contain word $w$ (note that a document can appear several times in ...


3

So, thank you for clarifying the question. Just to confirm that the question is asking how to set an appropriate threshold for face feature vectors (represented a a and b, for example). What I would recommend is to look at either cosine similarity or euclidean distance, which you have implemented. From here, I would then look the distribution fo the ...


3

First of all, I think you are confused with pretrained and finetuned. BERT is pretrained on a lot of text data. By using this pretrained BERT, you have a model that already have knowledge about text. BERT can then be finetuned on specific dataset, where BERT learn specific knowledge related to the dataset. That's why a finetuned BERT is bad on other ...


2

You can use the Universal Sentence Encoder from Google and calculate the similarity between texts using the cosine similarity or angular distance between their vector representations.


2

As mentioned in other answers, traditionally cosine is used to measure similarity between vectors whereas Levenshtein is used as a string similarity measure, i.e. measuring the distance between sequences of characters. Nevertheless they both can be used in non-traditional settings and are indeed comparable: the vectors compared with cosine can for instance ...


2

If I understand correctly, you're trying to map abstracts to their research papers. Here is a simple starting point: Compute a TF IDF model using the entire corpus (all abstracts + research papers). Use this model to transform your abstracts and research papers into a weighted vector representation. Under the TF IDF weighting scheme, these documents will ...


2

To summarize, you have labeled data in one class (positive) and unlabeled data. You want to find the positive examples in the unlabeled data. The general name for this setting in machine learing is one-class classification, which is a fairly broad field. A sub-area that is particularly relevant is positive-unlabeled learning, which is the problem of ...


2

One solution could be to: Get sentence embeddings from FastText Compute Euclidean Distance between the consecutive sentences If the distance between the consecutive sentences is close to 1, then, you may say the two sentences are talking about different topics. See here how to compute sentence embeddings for the English language: https://github.com/...


2

I think the answer(comment) of TwinPenguins provides you a good start. You could improve this by using a Metropolis Monte Carlo approach instead of just trying again and again generating new vectors. What you could do is: create a new vector with length similar to yours randomly sampled from the distribution of your interest and compute KLD$_{old}$ (cf. ...


1

So you want to identify a person via the similarity of the feature vector of the faces, with some database of known people, right? The similarity measures you said will help you identify the person not evaluate the outcome of that identification. To do this you need a set of people, who you know (i.e. are labelled). Then you need to perform your methodology: ...


1

Use a similarity vector with the dimension of all possible similarities, which initially are set to zero everywhere. Go over the shared images of one user and for each image set the vector to 1 at the positions of the similarities from the image. Do the same for another user. Use the scalar product as similarity metric between the two users. You can add ...


1

This corresponds to an NLP task called paraphrase detection. It's an active area of research, as far as I know there's no ready-to-use system able to perform this task very well, but there are probably a good few methods and prototypes around. A quick search gives these links for example: https://aclweb.org/aclwiki/Paraphrase_Identification_(...


1

About the data, you can use any question answering (QA) dataset to build your training and validation datasets. SQuAD is probably the most popular now, but there are many others (see here). From these datasets, you will be able to create a collection of pairs to question and their correct answers and questions and the wrong answers. About the model, BERT is ...


1

So, I gathered that your question is about how to evaluate whether a sentence A is a response to a given question Q or not. Drawing from knowledge of question-answer systems, you can evaluate whether a sentence responds to question Q by comparing the underlying lambda calculus (essentially translates language meaning into a logical, more computation-looking ...


1

That is a data mining problem, specifically affinity analysis. One common method to solve it is the Apriori algorithm.


1

Outlier detection doesn't sound like the most promising approach to me, as you have a model for the data. Some ideas you could try: use a hypothesis test to check the hypothesis that the stress values fit iid Gaussian with pre-defined standard deviation and unknown mean; use linear regression to fit a line that predicts stress as a function of time, and see ...


1

The first one is for computing the similarity between objects considering their representations as vectors. The second one is for computing the similarity between sequences of characters.


1

Consider the R package poLCA. Given a set of people with attributes that have categorical values, it can cluster the set into a fixed number of groups. As Anony-Mousse says above, it's not quite the same problem. Similarity would be to find people that are close to the one in the query. Clustering is close to answering that question for everybody. The name ...


1

You basically have data with a panel structure ($i=$shops, $t=$ weeks). So for a simple clustering problem, you need to "tell" the algorithm that observations for each $i$ (shops) are not independent. I think you are on the right way to treat each week $t$ as an own variable/feature here. In this case you would have 416 features and 1000 observations. This ...


1

The DBSCAN clustering algorithm has a built-in Jaccard distance metric. from sklearn.cluster import DBSCAN db = DBSCAN( metric='jaccard' ).fit(X) labels = db.labels_ # Number of clusters in labels, ignoring noise if present. n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0) Where X is your dataset with the related columns you want to use.


1

For quick proof of concept you could take pretrained embedder i.e. LASER. Here is unofficial pypi package. It works just fine. Though, keep in mind, embedders are meant for rather shorter chunks of texts. It makes little sense to assign single semantic meaning to more than few sentences. Embedder produces numerical vector. Once you've embedded lyrics from ...


1

Universal encoding and TF-IDF are two different beasts. I assume you mean the Vector Space Model transformed by TF-IDF. Either way: Neither tell you directly what the similarity of two texts are. Usually You'll use something like cosine distance to do that. For the VSM there are scores of techniques to transform it. To name a few: Rocchio Transformation, ...


1

I can give you some hint of doing so with deep learning approaches. It's easy to use gensim and sklearn python libraries. First, you need to extract the word embeddings which are vector of numbers to represent a word, and then take the average of the words within a sentence is a way of fining that vector representation for your sentence. So extract ...


1

This blog has the solution for short text similarity. They mainly use the BERT neural network model to find similarities between sentences. https://medium.com/@vimald8959/sentence-categorisation-short-text-similarity-61bb88fae15e


1

I think you are missing the model.infer_vector(new_sentence). You need to infer the new vectors based on your trained model. You can find more details in Assessing the Model section here. The similarity is between vectors but not the normalised tokens. So, you have to infer them first using your model.


1

You have to Build Term Matrix with TF-IDF and N-Grams. Once the matrix is build then you have to calculate the proximity between string and based on proximity you have to group together those strings.


Only top voted, non community-wiki answers of a minimum length are eligible