8
votes
Training Doc2Vec and Word2Vec at the same time
You need to be careful with the assumptions you make about the doc2vec implementation. Here are some useful concepts:
Word2vec has two different model implementations (Skip-gram and Continuous-bag-of-...
- 1,762
7
votes
Accepted
Text similarity with sentence embeddings
One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.
The ...
- 16.8k
7
votes
Accepted
How to measure the similarity between two text documents?
In general,there are two ways for finding document-document similarity
TF-IDF approach
Make a text corpus containing all words of documents . You have to use tokenisation and stop word removal . ...
- 116
4
votes
Accepted
Which algorithm Doc2Vec uses?
Word2Vec is not a combination of two models, rather both are variants of word2vec. Similarly doc2vec has Distributed Memory(DM) model and Distributed Bag of words (DBOW) model. Based on the context ...
- 1,924
3
votes
ways to represent document by its keyword vectors
If you have the vectors for your keywords, you can aggregate those to get the document vectors. The simplest(and probably one of the most effective) way is to average out your keyword vectors to form ...
- 781
3
votes
Accepted
Dynamic clustering for text documents
It sounds as if you don't need clustering.
But rather you are trying to detect near duplicates.
The difference is that clustering tries to organize everything with a focus on the larger, overall ...
- 7,769
2
votes
Accepted
Can I use euclidean distance for Latent Dirichlet Allocation document similarity?
Euclidean distance -by which in this application, I assume you mean the euclidean distance in an $n$-dimensional space defined by the distribution of document contents among $n$ topics considered, is ...
- 1,505
2
votes
Accepted
Finding similar articles in realtime
Elasticsearch is the right tool to use if you don't want to code this yourself. Indeed, you need an indexing algorithm that is able to efficiently retrieve pieces of texts in a big database, and SQL ...
- 1,295
2
votes
Dynamic clustering for text documents
As you say, thinking ahead about the number of clusters may be limiting. A simple solution would be to use KNN.
However, KNN can be pretty expensive to run over hundreds of ks of documents. In order ...
- 241
2
votes
ways to represent document by its keyword vectors
It sounds like you're trying to do some sort of topic modeling. I might recommend something like Latent Semantic Analysis (LSA), Latent Dirichlet allocation (LDA), or Latent Semantic Indexing (LSI). ...
- 351
2
votes
Text Similarities: which nlp methods to use?
There are many ways to see how texts are similar, but this will depend on your use case.
Semantics
Nowadays, word embeddings are getting popular. Like it was suggested in the comments, you could use ...
- 3,308
2
votes
How to measure the similarity between two text documents?
I would probably approach this by creating a set of features for each document, you could, for example, use a Bag of words representation or TF-IDF term weighting. Following this, you can compute the ...
- 166
2
votes
How to measure the similarity between two text documents?
The most common way is to measure the similarity between two text documents is distance in a vector space. A vector space model can be created by using word count, tf-idf, word embeddings, or document ...
- 16.8k
2
votes
Automatic code checking
You can start with MOSS (Measure Of Software Similarity). It can find similar software documents in a set of software documents. It has several nice properties - whitespace insensitivity, noise ...
- 16.8k
2
votes
Data wrangling for a big set of docx files advice!
The big problem with docx files is that they have a ton of content related to formatting that most people find irreverent when scraping docx files. Hence, one approach is to convert the docx files to ...
- 121
2
votes
Accepted
TS-SS and Cosine similarity among text documents using TF-IDF in Python
Normalised vectors have magnitude 1, so it doesn't matter if you explicitly divide by the magnitudes or not. It's mathematically equivalent either way.
I see no reason that you couldn't use ...
- 3,780
2
votes
Document similarity
I did something similar a while ago. We wanted to classify several types of pdf.
We first extracted the text of the documents.
We created NLP features with the text
Then added pdf metadata: size of ...
- 5,591
2
votes
Accepted
Fastest way for 1 vs all lookup on embeddings
There are libraries that are specialized in exactly that task, for instance FAISS by Facebook AI Research:
Faiss is a library for efficient similarity search and clustering of dense vectors. It ...
- 15.9k
2
votes
Accepted
Cosine vs Manhattan for Text Similarity
Intuitively, if you normalized the vectors before using them, or if they all ended up having almost unit norm after training, then a small $l_1$ norm will imply that the angle between the vectors is ...
- 355
1
vote
Accepted
pandas.isna() vs pandas.DataFrame.isna()
They call the same underlying method, so there is no functional difference.
Calling the dataframe member function is preferred for OOP patterns, but there are many redundancies/aliases in pandas and ...
- 2,422
1
vote
Hash function for text documents that maps similar documents to the same value
Hashing the unique copies of anything, including documents, is most commonly called fingerprinting.
Picking the fingerprinting hash function depends on your use case. For your use case, pick a ...
- 16.8k
1
vote
Accepted
How to classify a document by image?
You might want to look at Siamese CNNs depending on the size of your dataset. A good introduction can be found here.
- 136
1
vote
Checking TF-IDF Results
Yes, Cosine TF-IDF is quite transparent so it's usually reasonably easy to visualize the words which contribute the most to a score. Cosine is defined as the dot product divided by the product of the ...
- 22.1k
1
vote
How to compute document similarities in case of source codes?
Why not use all non-code indicators as 'handprints?' For example, many IDE's will add specific comments to documents when they are started. Also, and this is one that would be easy to detect my work, ...
- 356
1
vote
How to implement a basic query management and recommendation system
You might want to take a look at the solr project to see if it is a good fit.
- 464
1
vote
Automatic code checking
An alternative approach which is probably not as advanced yet for this is some combination of a deep learning system with and modification of output to provide a 'similarity likelyhood.'
This paper ...
- 571
1
vote
Accepted
How to correctly infer vectors in Gensim doc2vec?
I would use the first approach, given that both train and test are known, there is no need of generalization i.e. you don't expect unseen vectors.
In order to avoid the problem you mentioned, you ...
- 2,156
1
vote
Accepted
Cosine similarity between query and document confusion
You want to use all of the terms in the vector.
In your example, where your query vector $\mathbf{q} = [0,1,0,1,1]$ and your document vector $\mathbf{d} = [1,1,1,0,0]$, the cosine similarity is ...
- 3,780
1
vote
What techniques should I use to compare the similarity between a bunch of texts?
As of now, I can think of two ways to formulate this problem:
1. Search problem
Parse your job listings and index them in some sort of search engine like Solr or ElasticSearch. You can build ...
- 406
1
vote
Which algorithm Doc2Vec uses?
Distributed Memory model preserves the word order in a document whereas Distributed Bag of words just uses the bag of words approach, which doesn't preserve any word order.
This has been explained in ...
Only top scored, non community-wiki answers of a minimum length are eligible