8 votes

Training Doc2Vec and Word2Vec at the same time

You need to be careful with the assumptions you make about the doc2vec implementation. Here are some useful concepts: Word2vec has two different model implementations (Skip-gram and Continuous-bag-of-...
user avatar
  • 1,762
7 votes
Accepted

Text similarity with sentence embeddings

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector. The ...
user avatar
7 votes
Accepted

How to measure the similarity between two text documents?

In general,there are two ways for finding document-document similarity TF-IDF approach Make a text corpus containing all words of documents . You have to use tokenisation and stop word removal . ...
user avatar
4 votes
Accepted

Which algorithm Doc2Vec uses?

Word2Vec is not a combination of two models, rather both are variants of word2vec. Similarly doc2vec has Distributed Memory(DM) model and Distributed Bag of words (DBOW) model. Based on the context ...
user avatar
  • 1,924
3 votes

ways to represent document by its keyword vectors

If you have the vectors for your keywords, you can aggregate those to get the document vectors. The simplest(and probably one of the most effective) way is to average out your keyword vectors to form ...
user avatar
3 votes
Accepted

Dynamic clustering for text documents

It sounds as if you don't need clustering. But rather you are trying to detect near duplicates. The difference is that clustering tries to organize everything with a focus on the larger, overall ...
user avatar
2 votes
Accepted

Can I use euclidean distance for Latent Dirichlet Allocation document similarity?

Euclidean distance -by which in this application, I assume you mean the euclidean distance in an $n$-dimensional space defined by the distribution of document contents among $n$ topics considered, is ...
user avatar
2 votes
Accepted

Finding similar articles in realtime

Elasticsearch is the right tool to use if you don't want to code this yourself. Indeed, you need an indexing algorithm that is able to efficiently retrieve pieces of texts in a big database, and SQL ...
user avatar
  • 1,295
2 votes

Dynamic clustering for text documents

As you say, thinking ahead about the number of clusters may be limiting. A simple solution would be to use KNN. However, KNN can be pretty expensive to run over hundreds of ks of documents. In order ...
user avatar
  • 241
2 votes

ways to represent document by its keyword vectors

It sounds like you're trying to do some sort of topic modeling. I might recommend something like Latent Semantic Analysis (LSA), Latent Dirichlet allocation (LDA), or Latent Semantic Indexing (LSI). ...
user avatar
  • 351
2 votes

Text Similarities: which nlp methods to use?

There are many ways to see how texts are similar, but this will depend on your use case. Semantics Nowadays, word embeddings are getting popular. Like it was suggested in the comments, you could use ...
user avatar
2 votes

How to measure the similarity between two text documents?

I would probably approach this by creating a set of features for each document, you could, for example, use a Bag of words representation or TF-IDF term weighting. Following this, you can compute the ...
user avatar
2 votes

How to measure the similarity between two text documents?

The most common way is to measure the similarity between two text documents is distance in a vector space. A vector space model can be created by using word count, tf-idf, word embeddings, or document ...
user avatar
2 votes

Automatic code checking

You can start with MOSS (Measure Of Software Similarity). It can find similar software documents in a set of software documents. It has several nice properties - whitespace insensitivity, noise ...
user avatar
2 votes

Data wrangling for a big set of docx files advice!

The big problem with docx files is that they have a ton of content related to formatting that most people find irreverent when scraping docx files. Hence, one approach is to convert the docx files to ...
user avatar
  • 121
2 votes
Accepted

TS-SS and Cosine similarity among text documents using TF-IDF in Python

Normalised vectors have magnitude 1, so it doesn't matter if you explicitly divide by the magnitudes or not. It's mathematically equivalent either way. I see no reason that you couldn't use ...
user avatar
  • 3,780
2 votes

Document similarity

I did something similar a while ago. We wanted to classify several types of pdf. We first extracted the text of the documents. We created NLP features with the text Then added pdf metadata: size of ...
user avatar
2 votes
Accepted

Fastest way for 1 vs all lookup on embeddings

There are libraries that are specialized in exactly that task, for instance FAISS by Facebook AI Research: Faiss is a library for efficient similarity search and clustering of dense vectors. It ...
user avatar
  • 15.9k
2 votes
Accepted

Cosine vs Manhattan for Text Similarity

Intuitively, if you normalized the vectors before using them, or if they all ended up having almost unit norm after training, then a small $l_1$ norm will imply that the angle between the vectors is ...
user avatar
  • 355
1 vote
Accepted

pandas.isna() vs pandas.DataFrame.isna()

They call the same underlying method, so there is no functional difference. Calling the dataframe member function is preferred for OOP patterns, but there are many redundancies/aliases in pandas and ...
user avatar
  • 2,422
1 vote

Hash function for text documents that maps similar documents to the same value

Hashing the unique copies of anything, including documents, is most commonly called fingerprinting. Picking the fingerprinting hash function depends on your use case. For your use case, pick a ...
user avatar
1 vote
Accepted

How to classify a document by image?

You might want to look at Siamese CNNs depending on the size of your dataset. A good introduction can be found here.
user avatar
  • 136
1 vote

Checking TF-IDF Results

Yes, Cosine TF-IDF is quite transparent so it's usually reasonably easy to visualize the words which contribute the most to a score. Cosine is defined as the dot product divided by the product of the ...
user avatar
  • 22.1k
1 vote

How to compute document similarities in case of source codes?

Why not use all non-code indicators as 'handprints?' For example, many IDE's will add specific comments to documents when they are started. Also, and this is one that would be easy to detect my work, ...
user avatar
1 vote

How to implement a basic query management and recommendation system

You might want to take a look at the solr project to see if it is a good fit.
user avatar
1 vote

Automatic code checking

An alternative approach which is probably not as advanced yet for this is some combination of a deep learning system with and modification of output to provide a 'similarity likelyhood.' This paper ...
user avatar
1 vote
Accepted

How to correctly infer vectors in Gensim doc2vec?

I would use the first approach, given that both train and test are known, there is no need of generalization i.e. you don't expect unseen vectors. In order to avoid the problem you mentioned, you ...
user avatar
  • 2,156
1 vote
Accepted

Cosine similarity between query and document confusion

You want to use all of the terms in the vector. In your example, where your query vector $\mathbf{q} = [0,1,0,1,1]$ and your document vector $\mathbf{d} = [1,1,1,0,0]$, the cosine similarity is ...
user avatar
  • 3,780
1 vote

What techniques should I use to compare the similarity between a bunch of texts?

As of now, I can think of two ways to formulate this problem: 1. Search problem Parse your job listings and index them in some sort of search engine like Solr or ElasticSearch. You can build ...
user avatar
  • 406
1 vote

Which algorithm Doc2Vec uses?

Distributed Memory model preserves the word order in a document whereas Distributed Bag of words just uses the bag of words approach, which doesn't preserve any word order. This has been explained in ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible