Questions tagged [similar-documents]
The similar-documents tag has no usage guidance.
24
questions with no upvoted or accepted answers
3
votes
2
answers
73
views
Gensim doc2vec error: KeyError: "word 'senseless' not in vocabulary"
I am new to machine learning and tried doc2vec on quora duplicate dataset. new_dfx has columns 'question1' and 'question2' which has preprocessed questions in each row. Following is the tagged ...
2
votes
0
answers
48
views
Preprocessing for Document Similarity Using Doc2Vec
I'm trying to determine document similarity using Doc2Vec on a large series of legal opinions, which can contain some highly jargonistic language and phrases (e.g. en banc, de novo, etc.). I'm ...
- 21
2
votes
0
answers
391
views
Unsupervised document similarity state of the art
I have a set of N documents with lengths ranging from 0 to more than 20000 characters. I want to calculate a similarity score between 0 and 1 between all pairs of documents where a higher number ...
- 21
2
votes
2
answers
136
views
Semantic Search
There is a problem we are trying to solve where we want to do semantic search on our set of data,
i.e we have a domain specific data (example: sentences talking about automobiles)
Our data is just a ...
2
votes
1
answer
265
views
How to implement Semantic Search in R or Python
I have a task to provide semantic searching capabilities. For example, if I have a dataset of resume and if I search for "machine learning" than it should return me all resumes which have data science-...
- 21
1
vote
1
answer
826
views
Classification of scanned documents in pdf files using deep learning or NLP
I know classifying images using cnn but I have a problem where I have multiple types of scanned documents in a pdf file on different pages. Some types of scanned documents present in multiple pages ...
- 11
1
vote
1
answer
86
views
Document ranking on a web scraped dataset without any labelled data
I want to create a document ranking model which returns similar rows in the dataset for a sample query. The text in this corpus is standard english but without any labels (ie no query-related ...
- 11
1
vote
0
answers
144
views
Performing actual deduplication using LSH
I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
- 121
1
vote
0
answers
116
views
Matching documents from different sets with tfidf and cosine distance
I have two different set of documents S1, S2, with 30 text documents each.
Using some text representation method, such as tfidf ...
- 113
1
vote
0
answers
45
views
Extracting document templates from similar documents
Using very basic techniques (zone segmentation + OPTICS) I was able to organize a set of around 10^4 business documents (invoices, receipts) into hierarchy of clusters of documents of similar layout. ...
- 687
1
vote
0
answers
1k
views
How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?
I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data.
At present I ...
- 11
1
vote
0
answers
31
views
What is the best technique to transform documents into vectors?
What is the best algorithm between doc2vec and Singular Value Decomposition (SVD) to transform a set of 600 documents of around 1000 words each into vectors ?
- 567
1
vote
0
answers
589
views
Doc2vec most similar document to a query string
I'm working on a project and I created doc2vec representation of different academics which include their patents and publications etc. For each publication and patent I have information such as title ...
- 11
1
vote
0
answers
119
views
Counting similarity between two GIT commits
I am writing as part of my master thesis work an analysis of developers' work. I am looking for a new parameters to be consumed in Topological Data Analysis process. I would like to compare a ...
- 11
1
vote
0
answers
103
views
What do I️ do with an array of log probabilities? Doc2vec
I️ am working on a document classifier utilizing the Gensim libabry (doc2vec). After creating the model, I️ run the doc2vec.score() function on a document collection per document.
According to the ...
- 11
0
votes
0
answers
7
views
how to find documents similar to a predefined set of documents
From big population of documents I would like to find those similar to a predefined set of documents.
All documents inside the set are similar to each other, but very few documents from the population ...
0
votes
0
answers
32
views
LSA Model Improvement
I followed gensim's Core Tutorial and build an LSA Classification, topic modeling and Document Similarity model for newsgroups dataset.
My code is available here.
I need help with below 3 concepts.
...
- 1
0
votes
0
answers
109
views
Looking for more recent dataset for document classfication
I am trying to develop an NLP - CNN algorithm to detect documents with sensitive information such as passport, license and distinguish them from other documents like resume, email, form or ...
- 11
0
votes
1
answer
179
views
Is it possible to classify documents of corpus using labels?
I have a corpus of 23000 documents that need to be classified into 5 different categories. I do not have any labeled data available to me, just freeform text documents and labels(yes, one-word labels, ...
0
votes
2
answers
46
views
Scalable way to group users with similar titles purchased
I'm trying to figure out the best way to group customers based on checkout items in their shopping cart. I have the basket, and what's in the basket, but am at a complete loss on how to group all the ...
- 101
0
votes
1
answer
1k
views
How to do template matching without opencv?
How to do template matching without OpenCV?
I have an order invoice of documents belonging to Amazon, eBay, Flipkart, SnapDeal, and I want to extract less information from the order invoice.
Since the ...
- 1,361
0
votes
1
answer
517
views
Length of document in doc2vec
I have 100 sentences that I want to cluster based on similarity. I've used doc2vec to vectorize the sentences into 20 dimensional vectors and applied kmeans to cluster them. I haven't got the desired ...
- 229
0
votes
0
answers
100
views
Incorporating new features in document similarity task
I have a model pipeline for finding similar text documents given an input query text. The model is very simple; I have a corpus of documents on which I train a TfIDF model. When a query is input, we ...
- 1,564
-1
votes
1
answer
2k
views
How to get relevancy score of a term with respect to text/document
I am working on the literature documents. I am able to identify important entities using NER and Ontologies. Now I will like to assign the relevance score to the identified entities with respect to ...
- 99