Questions tagged [similar-documents]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
7
votes
0answers
557 views

What is the difference between fasttext and DANs in document classification?

I came across two interesting papers which describe promising approaches for document classification using word embedding. 1. The fasttext algorithm Described in the paper Bag of Tricks for ...
5
votes
1answer
152 views

How to compute document similarities in case of source codes?

I try to detect the probability of common authorship (person, company) of different kind of source code texts (webpages, program codes). My first idea is to apply the usual NLP tools like any token ...
4
votes
2answers
117 views

Automatic code checking

I have some experience in machine learning, mainly clustering and classifiers. However, I am somewhat of a newbie when it comes to NLP. That said I am aware of all the various issues and ...
4
votes
1answer
403 views

Using Spark for finding similar users to a user?

I read about https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html but couldn't find a spark library for this implementation. I have columnar string dataset. I have a dataset ...
3
votes
1answer
2k views

Training Doc2Vec and Word2Vec at the same time

As far as I can tell the typical Doc2Vec implementation (e.g. Gensim) first trains the word vectors and afterwards the document vectors were the word vectors are fixed. If my goal is that ...
3
votes
1answer
745 views

Can I use euclidean distance for Latent Dirichlet Allocation document similarity?

I have a Latent Dirichlet Allocation (LDA) model with $K$ topics trained on a corpus with $M$ documents. Due to my hyper parameter configurations, the output topic distributions for each document is ...
2
votes
3answers
472 views

How to measure the similarity between two text documents?

Assume, I have 100 text documents, and I want to cluster those documents. The first step is the construct pairwise similarity matrix 100X100 for the documents My ...
2
votes
2answers
46 views

ways to represent document by its keyword vectors

I have documents, say for example, D1, D2, D3... Dm. Every Di has its individual components or keywords k1, k2, k3,... kn, where ki is an n-dimensional vector. The number of individual components ...
2
votes
3answers
1k views

Which algorithm Doc2Vec uses?

Like Word2vec is not a single algorithm but combination of two, namely, CBOW and Skip-Gram model; is Doc2Vec also a combination of any such algorithms? Or is it an algorithm in itself?
2
votes
1answer
35 views

How to classify a document by image? [closed]

I need an opens source solution to classify a document. I do not want to use NLP i need only to check the look and feel. I tried OpenCV. I have a template and i need to match it. ...
2
votes
0answers
25 views

Data wrangling for a big set of docx files advice!

I'm looking for some advice on a data wrangling problem I'm trying to solve. I've spent a week solid taking different approaches and nothing seems to be quite perfect. Just FYI, this is my first big (...
2
votes
0answers
2k views

Doc2vec to calculate cosine similarity - absolutely inaccurate

I'm trying to modify the Doc2vec tutorial to calculate cosine similarity and take Pandas dataframes instead of .txt documents. I ...
1
vote
2answers
219 views

Dynamic clustering for text documents

I have few hundred thousands of text documents. Some of them are pretty similar - they differ just in ex. names or some numbers, all other text is the same. I would like to cluster these documents, so ...
1
vote
1answer
21 views

Checking TF-IDF Results

I am working with TF-IDF and cosine similarity to do document comparisons and given a document, which document in the data is the most similar. However, sometimes it returns a high similarity between ...
1
vote
1answer
3k views

Cosine similarity between query and document confusion

I am going through the Manning book for Information retrieval. Currently I am at the part about cosine similarity. One thing is not clear for me. Let's say that I have the tf idf vectors for the ...
1
vote
2answers
893 views

What techniques should I use to compare the similarity between a bunch of texts?

If I have a list of job postings stored as raw texts and I want to compare the similarity of all the job postings to a given resume, what techniques or algorithms should I use? I'm thinking of ...
1
vote
0answers
22 views

What is the best technique to transform documents into vectors?

What is the best algorithm between doc2vec and Singular Value Decomposition (SVD) to transform a set of 600 documents of around 1000 words each into vectors ?
1
vote
0answers
108 views

How to implement Semantic Search in R or Python

I have a task to provide semantic searching capabilities. For example, if I have a dataset of resume and if I search for "machine learning" than it should return me all resumes which have data science-...
1
vote
0answers
144 views

Doc2vec most similar document to a query string

I'm working on a project and I created doc2vec representation of different academics which include their patents and publications etc. For each publication and patent I have information such as title ...
1
vote
0answers
73 views

Counting similarity between two GIT commits

I am writing as part of my master thesis work an analysis of developers' work. I am looking for a new parameters to be consumed in Topological Data Analysis process. I would like to compare a ...
1
vote
0answers
82 views

Incorporating new features in document similarity task

I have a model pipeline for finding similar text documents given an input query text. The model is very simple; I have a corpus of documents on which I train a TfIDF model. When a query is input, we ...
1
vote
0answers
60 views

What do I️ do with an array of log probabilities? Doc2vec

I️ am working on a document classifier utilizing the Gensim libabry (doc2vec). After creating the model, I️ run the doc2vec.score() function on a document collection per document. According to the ...
0
votes
1answer
2k views

Text Similarities: which nlp methods to use?

I have data where there is text for each user A visiting a business B. I want to find similarity between each user using their text. Question 1: Which NLP method should I start with? I have tried ...
0
votes
1answer
406 views

Finding similar articles in realtime [closed]

I want to build a large document (news article) searchable database, such as when adding a new article I will be able to quickly find X most similar articles from it. What is the right tech/algorithm/...
0
votes
1answer
24 views

How to implement a basic query management and recommendation system

I'm trying to prototype a system where given a textual query (e.g. a question), I get a list of most relevant documents/questions among a pool of available documents/questions (similar to what we see ...
0
votes
1answer
2k views

How to correctly infer vectors in Gensim doc2vec?

I would like to know which is the correct procedure for inferring vectors in Gensim doc2vec. I have a dataframe df with a feature, called ...
0
votes
1answer
1k views

How to improve the accuracy of a Doc2Vec model (Gensim) in case of a toy-sized data set?

I'm building an NLP question-answering application using Doc2Vec technique in gensim package of Python. My training questions is very small, only 20 documents and I ...
0
votes
1answer
15 views

Length of document in doc2vec

I have 100 sentences that I want to cluster based on similarity. I've used doc2vec to vectorize the sentences into 20 dimensional vectors and applied kmeans to cluster them. I haven't got the desired ...
0
votes
0answers
5 views

Training data for doc2vec models, general vs specific

I have quite a general question about doc2vec models. Let's say I have a specific NLP task whose goal is to understand the similarity between two sports news articles. Now I have the option to train ...
0
votes
0answers
66 views

How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?

I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data. At present I ...
0
votes
0answers
186 views

BERT or ELMo for Document Similarity

Does anyone use BERT or ELMo language models to determine the similarity between two text documents? My question aims to collect all possible ways for combining the contextual word embeddings ...
0
votes
0answers
37 views

Doc2vec model.docvecs giving varying output

I am using doc2vec to vectorize input text. I am converting my input dataset to tagged data and giving it as input. Initially I tried with a data of 27 input text: ...
0
votes
0answers
14 views

how to get similary of unseen document with whole corpus

I have a trained lda model.now i want to find the new unseen document similarity with the corpus. ...
0
votes
0answers
154 views

Normalizing Jaccard similarity scores in relation to differences in document length

The Jaccard similarity of two documents A and B can be defined as the size of their intersection (how many tokens are in both docs) divided by the size of their union (total number of tokens found in ...
-1
votes
1answer
831 views

How to get relevancy score of a term with respect to text/document

I am working on the literature documents. I am able to identify important entities using NER and Ontologies. Now I will like to assign the relevance score to the identified entities with respect to ...