Questions tagged [similar-documents]

The tag has no usage guidance.

22 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3 votes
2 answers
720 views

Gensim doc2vec error: KeyError: "word 'senseless' not in vocabulary"

I am new to machine learning and tried doc2vec on quora duplicate dataset. new_dfx has columns 'question1' and 'question2' which has preprocessed questions in each row. Following is the tagged ...
Ankit Rohilla's user avatar
2 votes
0 answers
58 views

Preprocessing for Document Similarity Using Doc2Vec

I'm trying to determine document similarity using Doc2Vec on a large series of legal opinions, which can contain some highly jargonistic language and phrases (e.g. en banc, de novo, etc.). I'm ...
user118648's user avatar
2 votes
0 answers
426 views

Unsupervised document similarity state of the art

I have a set of N documents with lengths ranging from 0 to more than 20000 characters. I want to calculate a similarity score between 0 and 1 between all pairs of documents where a higher number ...
user7017793's user avatar
2 votes
2 answers
212 views

Semantic Search

There is a problem we are trying to solve where we want to do semantic search on our set of data, i.e we have a domain specific data (example: sentences talking about automobiles) Our data is just a ...
Farhaan Bukhsh's user avatar
2 votes
0 answers
58 views

Extracting document templates from similar documents

Using very basic techniques (zone segmentation + OPTICS) I was able to organize a set of around 10^4 business documents (invoices, receipts) into hierarchy of clusters of documents of similar layout. ...
dzieciou's user avatar
  • 697
2 votes
2 answers
339 views

How to implement Semantic Search in R or Python

I have a task to provide semantic searching capabilities. For example, if I have a dataset of resume and if I search for "machine learning" than it should return me all resumes which have data science-...
Yash Kanojia's user avatar
1 vote
1 answer
1k views

Classification of scanned documents in pdf files using deep learning or NLP

I know classifying images using cnn but I have a problem where I have multiple types of scanned documents in a pdf file on different pages. Some types of scanned documents present in multiple pages ...
Sherlock 's user avatar
1 vote
0 answers
169 views

Performing actual deduplication using LSH

I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
Marcin Zablocki's user avatar
1 vote
0 answers
171 views

Matching documents from different sets with tfidf and cosine distance

I have two different set of documents S1, S2, with 30 text documents each. Using some text representation method, such as tfidf ...
lynx's user avatar
  • 113
1 vote
0 answers
1k views

How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?

I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data. At present I ...
Vikrant's user avatar
  • 11
1 vote
0 answers
36 views

What is the best technique to transform documents into vectors?

What is the best algorithm between doc2vec and Singular Value Decomposition (SVD) to transform a set of 600 documents of around 1000 words each into vectors ?
Alexis Pister's user avatar
1 vote
0 answers
628 views

Doc2vec most similar document to a query string

I'm working on a project and I created doc2vec representation of different academics which include their patents and publications etc. For each publication and patent I have information such as title ...
qiqi's user avatar
  • 11
1 vote
0 answers
132 views

Counting similarity between two GIT commits

I am writing as part of my master thesis work an analysis of developers' work. I am looking for a new parameters to be consumed in Topological Data Analysis process. I would like to compare a ...
MKMZ's user avatar
  • 11
1 vote
0 answers
106 views

What do I️ do with an array of log probabilities? Doc2vec

I️ am working on a document classifier utilizing the Gensim libabry (doc2vec). After creating the model, I️ run the doc2vec.score() function on a document collection per document. According to the ...
ldavila07's user avatar
0 votes
0 answers
35 views

LSA Model Improvement

I followed gensim's Core Tutorial and build an LSA Classification, topic modeling and Document Similarity model for newsgroups dataset. My code is available here. I need help with below 3 concepts. ...
Bala's user avatar
  • 1
0 votes
0 answers
124 views

Looking for more recent dataset for document classfication

I am trying to develop an NLP - CNN algorithm to detect documents with sensitive information such as passport, license and distinguish them from other documents like resume, email, form or ...
siya m's user avatar
  • 11
0 votes
1 answer
234 views

Is it possible to classify documents of corpus using labels?

I have a corpus of 23000 documents that need to be classified into 5 different categories. I do not have any labeled data available to me, just freeform text documents and labels(yes, one-word labels, ...
Uttakarsh Tikku's user avatar
0 votes
2 answers
58 views

Scalable way to group users with similar titles purchased

I'm trying to figure out the best way to group customers based on checkout items in their shopping cart. I have the basket, and what's in the basket, but am at a complete loss on how to group all the ...
travistravis's user avatar
0 votes
1 answer
2k views

How to do template matching without opencv?

How to do template matching without OpenCV? I have an order invoice of documents belonging to Amazon, eBay, Flipkart, SnapDeal, and I want to extract less information from the order invoice. Since the ...
star's user avatar
  • 1,461
0 votes
0 answers
101 views

Incorporating new features in document similarity task

I have a model pipeline for finding similar text documents given an input query text. The model is very simple; I have a corpus of documents on which I train a TfIDF model. When a query is input, we ...
PyRsquared's user avatar
  • 1,604
0 votes
1 answer
661 views

Length of document in doc2vec

I have 100 sentences that I want to cluster based on similarity. I've used doc2vec to vectorize the sentences into 20 dimensional vectors and applied kmeans to cluster them. I haven't got the desired ...
Aishwarya A R's user avatar
-1 votes
1 answer
2k views

How to get relevancy score of a term with respect to text/document

I am working on the literature documents. I am able to identify important entities using NER and Ontologies. Now I will like to assign the relevance score to the identified entities with respect to ...
NKS's user avatar
  • 99