Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [similar-documents]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
26 views

pandas.isna() vs pandas.DataFrame.isna()

I've seen the two documentation pages for pandas.isna() and pandas.DataFrame.isna() but the difference is still unclear to me. Could someone explain the difference to me using examples?
1
vote
0answers
8 views

Hash function for text documents that maps similar documents to the same value

I have a web site that process text documents (typically 10-100 pages) submitted by users. Each time a user submits a document, I'd like to store a hash of the document, but I'd like similar ...
5
votes
2answers
145 views

Text similarity with sentence embeddings

I'm trying to calculate similarity between texts with various lengths. My current approach is following: Using Universal Sentence Encoder, I convert text to a set of vectors. I average these vectors ...
2
votes
1answer
27 views

Document embedding vs locality sensitive hashing for document clustering

I would like to compare two methods: locality sensitivity hashing and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a ...
0
votes
1answer
20 views

Length of document in doc2vec

I have 100 sentences that I want to cluster based on similarity. I've used doc2vec to vectorize the sentences into 20 dimensional vectors and applied kmeans to cluster them. I haven't got the desired ...
5
votes
1answer
154 views

How to compute document similarities in case of source codes?

I try to detect the probability of common authorship (person, company) of different kind of source code texts (webpages, program codes). My first idea is to apply the usual NLP tools like any token ...
0
votes
0answers
5 views

Training data for doc2vec models, general vs specific

I have quite a general question about doc2vec models. Let's say I have a specific NLP task whose goal is to understand the similarity between two sports news articles. Now I have the option to train ...
1
vote
0answers
157 views

How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?

I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data. At present I ...
2
votes
1answer
37 views

How to classify a document by image? [closed]

I need an opens source solution to classify a document. I do not want to use NLP i need only to check the look and feel. I tried OpenCV. I have a template and i need to match it. ...
0
votes
0answers
293 views

BERT or ELMo for Document Similarity

Does anyone use BERT or ELMo language models to determine the similarity between two text documents? My question aims to collect all possible ways for combining the contextual word embeddings ...
2
votes
0answers
26 views

Data wrangling for a big set of docx files advice!

I'm looking for some advice on a data wrangling problem I'm trying to solve. I've spent a week solid taking different approaches and nothing seems to be quite perfect. Just FYI, this is my first big (...
1
vote
1answer
23 views

Checking TF-IDF Results

I am working with TF-IDF and cosine similarity to do document comparisons and given a document, which document in the data is the most similar. However, sometimes it returns a high similarity between ...
0
votes
0answers
49 views

Doc2vec model.docvecs giving varying output

I am using doc2vec to vectorize input text. I am converting my input dataset to tagged data and giving it as input. Initially I tried with a data of 27 input text: ...
7
votes
0answers
564 views

What is the difference between fasttext and DANs in document classification?

I came across two interesting papers which describe promising approaches for document classification using word embedding. 1. The fasttext algorithm Described in the paper Bag of Tricks for ...
1
vote
0answers
24 views

What is the best technique to transform documents into vectors?

What is the best algorithm between doc2vec and Singular Value Decomposition (SVD) to transform a set of 600 documents of around 1000 words each into vectors ?
0
votes
0answers
15 views

how to get similary of unseen document with whole corpus

I have a trained lda model.now i want to find the new unseen document similarity with the corpus. ...
1
vote
0answers
132 views

How to implement Semantic Search in R or Python

I have a task to provide semantic searching capabilities. For example, if I have a dataset of resume and if I search for "machine learning" than it should return me all resumes which have data science-...
2
votes
3answers
956 views

How to measure the similarity between two text documents?

Assume, I have 100 text documents, and I want to cluster those documents. The first step is the construct pairwise similarity matrix 100X100 for the documents My ...
1
vote
0answers
162 views

Doc2vec most similar document to a query string

I'm working on a project and I created doc2vec representation of different academics which include their patents and publications etc. For each publication and patent I have information such as title ...
2
votes
2answers
49 views

ways to represent document by its keyword vectors

I have documents, say for example, D1, D2, D3... Dm. Every Di has its individual components or keywords k1, k2, k3,... kn, where ki is an n-dimensional vector. The number of individual components ...
0
votes
0answers
184 views

Normalizing Jaccard similarity scores in relation to differences in document length

The Jaccard similarity of two documents A and B can be defined as the size of their intersection (how many tokens are in both docs) divided by the size of their union (total number of tokens found in ...
2
votes
0answers
2k views

Doc2vec to calculate cosine similarity - absolutely inaccurate

I'm trying to modify the Doc2vec tutorial to calculate cosine similarity and take Pandas dataframes instead of .txt documents. I ...
1
vote
0answers
75 views

Counting similarity between two GIT commits

I am writing as part of my master thesis work an analysis of developers' work. I am looking for a new parameters to be consumed in Topological Data Analysis process. I would like to compare a ...
0
votes
1answer
2k views

Text Similarities: which nlp methods to use?

I have data where there is text for each user A visiting a business B. I want to find similarity between each user using their text. Question 1: Which NLP method should I start with? I have tried ...
0
votes
1answer
25 views

How to implement a basic query management and recommendation system

I'm trying to prototype a system where given a textual query (e.g. a question), I get a list of most relevant documents/questions among a pool of available documents/questions (similar to what we see ...
4
votes
2answers
118 views

Automatic code checking

I have some experience in machine learning, mainly clustering and classifiers. However, I am somewhat of a newbie when it comes to NLP. That said I am aware of all the various issues and ...
-1
votes
1answer
896 views

How to get relevancy score of a term with respect to text/document

I am working on the literature documents. I am able to identify important entities using NER and Ontologies. Now I will like to assign the relevance score to the identified entities with respect to ...
1
vote
0answers
84 views

Incorporating new features in document similarity task

I have a model pipeline for finding similar text documents given an input query text. The model is very simple; I have a corpus of documents on which I train a TfIDF model. When a query is input, we ...
2
votes
3answers
1k views

Which algorithm Doc2Vec uses?

Like Word2vec is not a single algorithm but combination of two, namely, CBOW and Skip-Gram model; is Doc2Vec also a combination of any such algorithms? Or is it an algorithm in itself?
3
votes
1answer
2k views

Training Doc2Vec and Word2Vec at the same time

As far as I can tell the typical Doc2Vec implementation (e.g. Gensim) first trains the word vectors and afterwards the document vectors were the word vectors are fixed. If my goal is that ...
4
votes
1answer
411 views

Using Spark for finding similar users to a user?

I read about https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html but couldn't find a spark library for this implementation. I have columnar string dataset. I have a dataset ...
0
votes
1answer
2k views

How to correctly infer vectors in Gensim doc2vec?

I would like to know which is the correct procedure for inferring vectors in Gensim doc2vec. I have a dataframe df with a feature, called ...
3
votes
1answer
777 views

Can I use euclidean distance for Latent Dirichlet Allocation document similarity?

I have a Latent Dirichlet Allocation (LDA) model with $K$ topics trained on a corpus with $M$ documents. Due to my hyper parameter configurations, the output topic distributions for each document is ...
1
vote
0answers
61 views

What do I️ do with an array of log probabilities? Doc2vec

I️ am working on a document classifier utilizing the Gensim libabry (doc2vec). After creating the model, I️ run the doc2vec.score() function on a document collection per document. According to the ...
0
votes
1answer
417 views

Finding similar articles in realtime [closed]

I want to build a large document (news article) searchable database, such as when adding a new article I will be able to quickly find X most similar articles from it. What is the right tech/algorithm/...
1
vote
1answer
3k views

Cosine similarity between query and document confusion

I am going through the Manning book for Information retrieval. Currently I am at the part about cosine similarity. One thing is not clear for me. Let's say that I have the tf idf vectors for the ...
0
votes
1answer
1k views

How to improve the accuracy of a Doc2Vec model (Gensim) in case of a toy-sized data set?

I'm building an NLP question-answering application using Doc2Vec technique in gensim package of Python. My training questions is very small, only 20 documents and I ...
1
vote
2answers
920 views

What techniques should I use to compare the similarity between a bunch of texts?

If I have a list of job postings stored as raw texts and I want to compare the similarity of all the job postings to a given resume, what techniques or algorithms should I use? I'm thinking of ...
1
vote
2answers
227 views

Dynamic clustering for text documents

I have few hundred thousands of text documents. Some of them are pretty similar - they differ just in ex. names or some numbers, all other text is the same. I would like to cluster these documents, so ...