Questions tagged [similar-documents]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2 votes
2 answers
33 views

Gensim doc2vec error: KeyError: "word 'senseless' not in vocabulary"

I am new to machine learning and tried doc2vec on quora duplicate dataset. new_dfx has columns 'question1' and 'question2' which has preprocessed questions in each row. Following is the tagged ...
0 votes
0 answers
7 views

how to find documents similar to a predefined set of documents

From big population of documents I would like to find those similar to a predefined set of documents. All documents inside the set are similar to each other, but very few documents from the population ...
1 vote
1 answer
44 views

document similarity using LDA probabilities

Let us say I have a LDA model trained on a corpus of text. I would like to know, for a newly given document, which one from the corpus is closet to it. But, to do so, I want to use probabilities ...
1 vote
1 answer
31 views

Deep learning techniques for concept similarity?

Given a corpus of product descriptions (say, vacuum cleaners), I'm looking for a way to group the documents that are all of the same type (where a type can be ...
1 vote
1 answer
246 views

Building a graph out of a large text corpus

I'm given a large amount of documents upon which I should perform various kinds of analysis. Since the documents are to be used as a foundation of a final product, I thought about building a graph out ...
1 vote
1 answer
32 views

Document Similarity with User Preference

To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
1 vote
1 answer
713 views

Classification of scanned documents in pdf files using deep learning or NLP

I know classifying images using cnn but I have a problem where I have multiple types of scanned documents in a pdf file on different pages. Some types of scanned documents present in multiple pages ...
1 vote
1 answer
79 views

Document ranking on a web scraped dataset without any labelled data

I want to create a document ranking model which returns similar rows in the dataset for a sample query. The text in this corpus is standard english but without any labels (ie no query-related ...
  • 11
0 votes
1 answer
209 views

Cosine vs Manhattan for Text Similarity [closed]

I'm storing sentences in Elasticsearch as dense_vector field and used BERT for the embedding so each vector is 768 dim. ...
0 votes
0 answers
32 views

LSA Model Improvement

I followed gensim's Core Tutorial and build an LSA Classification, topic modeling and Document Similarity model for newsgroups dataset. My code is available here. I need help with below 3 concepts. ...
  • 1
1 vote
0 answers
138 views

Performing actual deduplication using LSH

I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
2 votes
0 answers
43 views

Preprocessing for Document Similarity Using Doc2Vec

I'm trying to determine document similarity using Doc2Vec on a large series of legal opinions, which can contain some highly jargonistic language and phrases (e.g. en banc, de novo, etc.). I'm ...
1 vote
0 answers
110 views

Matching documents from different sets with tfidf and cosine distance

I have two different set of documents S1, S2, with 30 text documents each. Using some text representation method, such as tfidf ...
  • 111
2 votes
0 answers
364 views

Unsupervised document similarity state of the art

I have a set of N documents with lengths ranging from 0 to more than 20000 characters. I want to calculate a similarity score between 0 and 1 between all pairs of documents where a higher number ...
0 votes
0 answers
105 views

Looking for more recent dataset for document classfication

I am trying to develop an NLP - CNN algorithm to detect documents with sensitive information such as passport, license and distinguish them from other documents like resume, email, form or ...
  • 11
1 vote
1 answer
39 views

Evaluation of recommendation systems

I have developed a content-based recommendation system and it is working fine. The input is a set of documents={d1,d2,d3,...,dn} and the output will be Top N similar documents for a given document ...
  • 159
1 vote
2 answers
168 views

Evaluate document similarity / content-based recommender system

I'm planning on building a basic content-based recommender system with word2vec and cosine similarity. The data consists of 300k documents in varying length. How do I evaluate my model if I have no ...
  • 143
1 vote
1 answer
233 views

What is considered short and long text in NLP (document similarity)

What is considered short and long text in NLP? I'm working on a dataset that contains documents from 10 to 600 words and I'm asking myself if I should treat them differently. Also, I haven't found a ...
  • 143
0 votes
1 answer
33 views

How to filter Items in Recommender Systems?

I have a Recommender System which recommends Articles based on Similarity from 3 Features, "Page-Title, Article Content, Tags". But some of the Articles are NSFW(Related to Adult Topics). I ...
  • 321
0 votes
1 answer
165 views

Is it possible to classify documents of corpus using labels?

I have a corpus of 23000 documents that need to be classified into 5 different categories. I do not have any labeled data available to me, just freeform text documents and labels(yes, one-word labels, ...
1 vote
0 answers
32 views

Document Similarity to List of Words in Sentiment Analysis [closed]

How would you go about finding document similarity to a list of words in Sentiment Analysis? Looking find document similarity to multiple lists of words in sentiment analysis. I had been working on ...
  • 111
0 votes
2 answers
45 views

Scalable way to group users with similar titles purchased

I'm trying to figure out the best way to group customers based on checkout items in their shopping cart. I have the basket, and what's in the basket, but am at a complete loss on how to group all the ...
1 vote
3 answers
2k views

Fastest way for 1 vs all lookup on embeddings

I have a dataset with about 1 000 000 texts where I have computed their sentence embeddings with a language model and stored them in a numpy array. I wish to compare a new unseen text to all the 1 ...
  • 163
2 votes
2 answers
126 views

Semantic Search

There is a problem we are trying to solve where we want to do semantic search on our set of data, i.e we have a domain specific data (example: sentences talking about automobiles) Our data is just a ...
0 votes
1 answer
1k views

How to do template matching without opencv?

How to do template matching without OpenCV? I have an order invoice of documents belonging to Amazon, eBay, Flipkart, SnapDeal, and I want to extract less information from the order invoice. Since the ...
  • 1,311
2 votes
2 answers
320 views

Document similarity

I have close to 50000 documents in plain text format. Is there a way in which I can group similar documents together? Similarity mostly here is the content similarity. Will transforming the text ...
  • 149
1 vote
0 answers
44 views

Extracting document templates from similar documents

Using very basic techniques (zone segmentation + OPTICS) I was able to organize a set of around 10^4 business documents (invoices, receipts) into hierarchy of clusters of documents of similar layout. ...
  • 687
5 votes
1 answer
785 views

TS-SS and Cosine similarity among text documents using TF-IDF in Python

A common way of calculating the cosine similarity between text based documents is to calculate tf-idf and then calculating the linear kernel of the tf-idf matrix. TF-IDF matrix is calculated using ...
  • 153
1 vote
1 answer
432 views

pandas.isna() vs pandas.DataFrame.isna()

I've seen the two documentation pages for pandas.isna() and pandas.DataFrame.isna() but the difference is still unclear to me. Could someone explain the difference to me using examples?
2 votes
2 answers
708 views

Hash function for text documents that maps similar documents to the same value

I have a web site that process text documents (typically 10-100 pages) submitted by users. Each time a user submits a document, I'd like to store a hash of the document, but I'd like similar ...
  • 121
2 votes
1 answer
1k views

Document embedding vs locality sensitive hashing for document clustering

I would like to compare two methods: locality sensitivity hashing and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a ...
7 votes
2 answers
5k views

Text similarity with sentence embeddings

I'm trying to calculate similarity between texts with various lengths. My current approach is following: Using Universal Sentence Encoder, I convert text to a set of vectors. I average these vectors ...
0 votes
1 answer
485 views

Length of document in doc2vec

I have 100 sentences that I want to cluster based on similarity. I've used doc2vec to vectorize the sentences into 20 dimensional vectors and applied kmeans to cluster them. I haven't got the desired ...
1 vote
0 answers
1k views

How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?

I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data. At present I ...
  • 11
2 votes
1 answer
135 views

How to classify a document by image? [closed]

I need an opens source solution to classify a document. I do not want to use NLP i need only to check the look and feel. I tried OpenCV. I have a template and i need to match it. ...
  • 23
4 votes
2 answers
468 views

Data wrangling for a big set of docx files advice!

I'm looking for some advice on a data wrangling problem I'm trying to solve. I've spent a week solid taking different approaches and nothing seems to be quite perfect. Just FYI, this is my first big (...
  • 41
1 vote
1 answer
75 views

Checking TF-IDF Results

I am working with TF-IDF and cosine similarity to do document comparisons and given a document, which document in the data is the most similar. However, sometimes it returns a high similarity between ...
1 vote
0 answers
30 views

What is the best technique to transform documents into vectors?

What is the best algorithm between doc2vec and Singular Value Decomposition (SVD) to transform a set of 600 documents of around 1000 words each into vectors ?
4 votes
3 answers
11k views

How to measure the similarity between two text documents?

Assume, I have 100 text documents, and I want to cluster those documents. The first step is the construct pairwise similarity matrix 100X100 for the documents My ...
  • 309
2 votes
1 answer
257 views

How to implement Semantic Search in R or Python

I have a task to provide semantic searching capabilities. For example, if I have a dataset of resume and if I search for "machine learning" than it should return me all resumes which have data science-...
1 vote
0 answers
582 views

Doc2vec most similar document to a query string

I'm working on a project and I created doc2vec representation of different academics which include their patents and publications etc. For each publication and patent I have information such as title ...
  • 11
2 votes
2 answers
330 views

ways to represent document by its keyword vectors

I have documents, say for example, D1, D2, D3... Dm. Every Di has its individual components or keywords k1, k2, k3,... kn, where ki is an n-dimensional vector. The number of individual components ...
  • 285
1 vote
0 answers
118 views

Counting similarity between two GIT commits

I am writing as part of my master thesis work an analysis of developers' work. I am looking for a new parameters to be consumed in Topological Data Analysis process. I would like to compare a ...
  • 11
1 vote
1 answer
5k views

Text Similarities: which nlp methods to use?

I have data where there is text for each user A visiting a business B. I want to find similarity between each user using their text. Question 1: Which NLP method should I start with? I have tried ...
0 votes
1 answer
34 views

How to implement a basic query management and recommendation system

I'm trying to prototype a system where given a textual query (e.g. a question), I get a list of most relevant documents/questions among a pool of available documents/questions (similar to what we see ...
  • 197
4 votes
2 answers
169 views

Automatic code checking

I have some experience in machine learning, mainly clustering and classifiers. However, I am somewhat of a newbie when it comes to NLP. That said I am aware of all the various issues and ...
0 votes
0 answers
100 views

Incorporating new features in document similarity task

I have a model pipeline for finding similar text documents given an input query text. The model is very simple; I have a corpus of documents on which I train a TfIDF model. When a query is input, we ...
  • 1,564
6 votes
1 answer
188 views

How to compute document similarities in case of source codes?

I try to detect the probability of common authorship (person, company) of different kind of source code texts (webpages, program codes). My first idea is to apply the usual NLP tools like any token ...
  • 8,177
3 votes
1 answer
3k views

Training Doc2Vec and Word2Vec at the same time

As far as I can tell the typical Doc2Vec implementation (e.g. Gensim) first trains the word vectors and afterwards the document vectors were the word vectors are fixed. If my goal is that ...
0 votes
1 answer
3k views

How to correctly infer vectors in Gensim doc2vec?

I would like to know which is the correct procedure for inferring vectors in Gensim doc2vec. I have a dataframe df with a feature, called ...
  • 683