8
You need to be careful with the assumptions you make about the doc2vec implementation. Here are some useful concepts:
Word2vec has two different model implementations (Skip-gram and Continuous-bag-of-words) and Doc2vec has analogous PV-DM and CBOW models:
Word2vec
Continuous-bag-of-words (CBOW)
Skip-gram
Doc2vec (Paragraph Vector)
Distributed Memory (PV-...
7
In general,there are two ways for finding document-document similarity
TF-IDF approach
Make a text corpus containing all words of documents . You have to use tokenisation and stop word removal . NLTK library provides all .
Convert the documents into tf-idf vectors .
Find the cosine-similarity between them or any new document for similarity measure.
You ...
6
One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.
The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to ...
4
Word2Vec is not a combination of two models, rather both are variants of word2vec. Similarly doc2vec has Distributed Memory(DM) model and Distributed Bag of words (DBOW) model. Based on the context words and the target word, these variants arised.
Note: the name of the model maybe confusing
Distriubted Bag of words is similar to Skip-gram model
...
3
If you have the vectors for your keywords, you can aggregate those to get the document vectors. The simplest(and probably one of the most effective) way is to average out your keyword vectors to form the document vector. Once you have the vectors for each of the documents, you can use similarity measures like cosine similarity to see how close your documents ...
3
It sounds as if you don't need clustering.
But rather you are trying to detect near duplicates.
The difference is that clustering tries to organize everything with a focus on the larger, overall structure. But much of your data probably isn't duplicate. Clustering is difficult and slow. Near duplicates is much easier, and much faster (e.g., with MinHash or ...
answered Mar 31 '17 at 20:35
Has QUIT--Anony-Mousse
7,5391 gold badge12 silver badges29 bronze badges
2
Euclidean distance -by which in this application, I assume you mean the euclidean distance in an $n$-dimensional space defined by the distribution of document contents among $n$ topics considered, is a valid measure to use in comparing the topics represented within two documents.
What you're doing by applying this method is quantifying a topic frequency ...
2
Elasticsearch is the right tool to use if you don't want to code this yourself. Indeed, you need an indexing algorithm that is able to efficiently retrieve pieces of texts in a big database, and SQL isn't particularly good at it. Moreover, Elasticsearch is quite user friendly, so it won't be an overkill to actually install it and use it. You might discover ...
2
As you say, thinking ahead about the number of clusters may be limiting. A simple solution would be to use KNN.
However, KNN can be pretty expensive to run over hundreds of ks of documents. In order to limit your search space, you should first filter out (and do it quickly) documents that there is no chance that would share a cluster with the new document. ...
2
It sounds like you're trying to do some sort of topic modeling. I might recommend something like Latent Semantic Analysis (LSA), Latent Dirichlet allocation (LDA), or Latent Semantic Indexing (LSI). LSA performs a matrix decomposition of the document x keyword occurrence matrix to extract salient topics. It can allow you to find the cosine similarity between ...
2
There are many ways to see how texts are similar, but this will depend on your use case.
Semantics
Nowadays, word embeddings are getting popular. Like it was suggested in the comments, you could use Doc2Vec to transform your sentence into a vector and calculate the cosine distance from each sentence.
The idea with these learned embeddings is that you kind ...
2
I would probably approach this by creating a set of features for each document, you could, for example, use a Bag of words representation or TF-IDF term weighting. Following this, you can compute the closeness of these features through some distance metric and use this as a measure of the similarity.
2
The most common way is to measure the similarity between two text documents is distance in a vector space. A vector space model can be created by using word count, tf-idf, word embeddings, or document embeddings. Distance is most often measured by cosine similarity. Once all the documents are in the same vector space, the 100x100 matrix with cosine ...
2
You can start with MOSS (Measure Of Software Similarity). It can find similar software documents in a set of software documents. It has several nice properties - whitespace insensitivity, noise suppression, and position independence. It is based on the more general idea of document fingerprinting where a hash is constructed of the document. The document hash ...
2
The big problem with docx files is that they have a ton of content related to formatting that most people find irreverent when scraping docx files. Hence, one approach is to convert the docx files to something more friendly like json that excludes all the irrelevant bits, and then use existing frameworks like JSON schema for pattern matching and jmespath ...
2
Normalised vectors have magnitude 1, so it doesn't matter if you explicitly divide by the magnitudes or not. It's mathematically equivalent either way.
I see no reason that you couldn't use normalised vectors in TS-SS, but it seems that the main motivation for using TS-SS in the first place is that it makes more sense for vectors that may have different ...
2
I did something similar a while ago. We wanted to classify several types of pdf.
We first extracted the text of the documents.
We created NLP features with the text
Then added pdf metadata: size of the file, number of pages, name of the document...
We then built a classification model with a few samples and did Active Learning
I guess that you could also ...
2
There are libraries that are specialized in exactly that task, for instance FAISS by Facebook AI Research:
Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and ...
2
Intuitively, if you normalized the vectors before using them, or if they all ended up having almost unit norm after training, then a small $l_1$ norm will imply that the angle between the vectors is small, hence the cosine similarity will be high. Conversely, almost colinear vectors will have almost equal coordinates because they all have unit length. So if ...
1
They call the same underlying method, so there is no functional difference.
Calling the dataframe member function is preferred for OOP patterns, but there are many redundancies/aliases in pandas and python in general.
In case you are curious, here is how the source code breaks down (it is a mess).
The DataFrame (pandas/core/frame.py) method is simply:
...
1
Hashing the unique copies of anything, including documents, is most commonly called fingerprinting.
Picking the fingerprinting hash function depends on your use case. For your use case, pick a rolling, non-cryptographic hash function. One of the simplest examples is Rabin–Karp algorithm. Once applied, similar documents will have similar hash values.
...
1
You might want to look at Siamese CNNs depending on the size of your dataset. A good introduction can be found here.
1
Yes, Cosine TF-IDF is quite transparent so it's usually reasonably easy to visualize the words which contribute the most to a score. Cosine is defined as the dot product divided by the product of the norms, so you can isolate the terms:
dotproduct(d_1,d_2) = tfidf(w1,d1) * tfidf(w1,d2) + tfidf(w2,d1) * tfidf(w2,d2) + ... + tfidf(wN,dN)
Ranking the words ...
1
Why not use all non-code indicators as 'handprints?' For example, many IDE's will add specific comments to documents when they are started. Also, and this is one that would be easy to detect my work, I have a tendency in python to copy entire import blocks from one script to another, whether or not I actually need all the imports or not. If someone uses a ...
1
You might want to take a look at the solr project to see if it is a good fit.
1
An alternative approach which is probably not as advanced yet for this is some combination of a deep learning system with and modification of output to provide a 'similarity likelyhood.'
This paper Code2vec, and this one have a neural network training on code to encode it as a vector in a scalable way in order to predict its properties, and then perform ...
1
I would use the first approach, given that both train and test are known, there is no need of generalization i.e. you don't expect unseen vectors.
In order to avoid the problem you mentioned, you have to find the most similar vector to a vector in test considering only vectors in train. For example:
train = [v1, v2, v3]
test = [v4, v5]
most_similar = {}
...
1
You want to use all of the terms in the vector.
In your example, where your query vector $\mathbf{q} = [0,1,0,1,1]$ and your document vector $\mathbf{d} = [1,1,1,0,0]$, the cosine similarity is computed as
similarity $= \frac{\mathbf{q} \cdot \mathbf{d}}{||\mathbf{q}||_2 ||\mathbf{d}||_2} = \frac{0\times1+1\times1+0\times1+1\times0+1\times0}{\sqrt{1^2+1^2+...
1
As of now, I can think of two ways to formulate this problem:
1. Search problem
Parse your job listings and index them in some sort of search engine like Solr or ElasticSearch. You can build capabilities like Semantic search using Word2Vec models, etc.
Now write a query engine which takes a resume and queries this Search engine. It will be blazing fast since ...
1
Distributed Memory model preserves the word order in a document whereas Distributed Bag of words just uses the bag of words approach, which doesn't preserve any word order.
This has been explained in details in this research paper.
Only top voted, non community-wiki answers of a minimum length are eligible