Skip to main content

Questions tagged [similar-documents]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
10 votes
1 answer
703 views

What is the difference between fasttext and DANs in document classification?

I came across two interesting papers that describe promising approaches for document classification using word embedding. 1. The fasttext algorithm Described in the paper Bag of Tricks for ...
user1043144's user avatar
8 votes
2 answers
6k views

Text similarity with sentence embeddings

I'm trying to calculate similarity between texts with various lengths. My current approach is following: Using Universal Sentence Encoder, I convert text to a set of vectors. I average these vectors ...
Kertis van Kertis's user avatar
6 votes
1 answer
2k views

Can I use euclidean distance for Latent Dirichlet Allocation document similarity?

I have a Latent Dirichlet Allocation (LDA) model with $K$ topics trained on a corpus with $M$ documents. Due to my hyper parameter configurations, the output topic distributions for each document is ...
PyRsquared's user avatar
  • 1,604
6 votes
1 answer
192 views

How to compute document similarities in case of source codes?

I try to detect the probability of common authorship (person, company) of different kind of source code texts (webpages, program codes). My first idea is to apply the usual NLP tools like any token ...
Hendrik's user avatar
  • 8,617
5 votes
1 answer
1k views

TS-SS and Cosine similarity among text documents using TF-IDF in Python

A common way of calculating the cosine similarity between text based documents is to calculate tf-idf and then calculating the linear kernel of the tf-idf matrix. TF-IDF matrix is calculated using ...
kgkmeekg's user avatar
  • 153
5 votes
1 answer
600 views

Using Spark for finding similar users to a user?

I read about https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html but couldn't find a spark library for this implementation. I have columnar string dataset. I have a dataset ...
Nikhil Verma's user avatar
4 votes
3 answers
12k views

How to measure the similarity between two text documents?

Assume, I have 100 text documents, and I want to cluster those documents. The first step is the construct pairwise similarity matrix 100X100 for the documents My ...
jason's user avatar
  • 329
4 votes
2 answers
176 views

Automatic code checking

I have some experience in machine learning, mainly clustering and classifiers. However, I am somewhat of a newbie when it comes to NLP. That said I am aware of all the various issues and ...
user2948208's user avatar
4 votes
2 answers
680 views

Data wrangling for a big set of docx files advice!

I'm looking for some advice on a data wrangling problem I'm trying to solve. I've spent a week solid taking different approaches and nothing seems to be quite perfect. Just FYI, this is my first big (...
mess1n's user avatar
  • 41
3 votes
1 answer
3k views

Training Doc2Vec and Word2Vec at the same time

As far as I can tell the typical Doc2Vec implementation (e.g. Gensim) first trains the word vectors and afterwards the document vectors were the word vectors are fixed. If my goal is that ...
Markus RH's user avatar
3 votes
3 answers
2k views

Which algorithm Doc2Vec uses?

Like Word2vec is not a single algorithm but combination of two, namely, CBOW and Skip-Gram model; is Doc2Vec also a combination of any such algorithms? Or is it an algorithm in itself?
Kshitiz 's user avatar
3 votes
2 answers
778 views

Gensim doc2vec error: KeyError: "word 'senseless' not in vocabulary"

I am new to machine learning and tried doc2vec on quora duplicate dataset. new_dfx has columns 'question1' and 'question2' which has preprocessed questions in each row. Following is the tagged ...
Ankit Rohilla's user avatar
2 votes
2 answers
452 views

ways to represent document by its keyword vectors

I have documents, say for example, D1, D2, D3... Dm. Every Di has its individual components or keywords k1, k2, k3,... kn, where ki is an n-dimensional vector. The number of individual components ...
Van Peer's user avatar
  • 285
2 votes
2 answers
356 views

Document similarity

I have close to 50000 documents in plain text format. Is there a way in which I can group similar documents together? Similarity mostly here is the content similarity. Will transforming the text ...
praneeth's user avatar
  • 149
2 votes
2 answers
1k views

Hash function for text documents that maps similar documents to the same value

I have a web site that process text documents (typically 10-100 pages) submitted by users. Each time a user submits a document, I'd like to store a hash of the document, but I'd like similar ...
minou's user avatar
  • 121
2 votes
1 answer
2k views

Document embedding vs locality sensitive hashing for document clustering

I would like to compare two methods: locality sensitivity hashing and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a ...
Robin Nicole's user avatar
2 votes
1 answer
211 views

How to classify a document by image? [closed]

I need an opens source solution to classify a document. I do not want to use NLP i need only to check the look and feel. I tried OpenCV. I have a template and i need to match it. ...
myml's user avatar
  • 23
2 votes
0 answers
61 views

Preprocessing for Document Similarity Using Doc2Vec

I'm trying to determine document similarity using Doc2Vec on a large series of legal opinions, which can contain some highly jargonistic language and phrases (e.g. en banc, de novo, etc.). I'm ...
user118648's user avatar
2 votes
0 answers
428 views

Unsupervised document similarity state of the art

I have a set of N documents with lengths ranging from 0 to more than 20000 characters. I want to calculate a similarity score between 0 and 1 between all pairs of documents where a higher number ...
user7017793's user avatar
2 votes
2 answers
217 views

Semantic Search

There is a problem we are trying to solve where we want to do semantic search on our set of data, i.e we have a domain specific data (example: sentences talking about automobiles) Our data is just a ...
Farhaan Bukhsh's user avatar
2 votes
0 answers
61 views

Extracting document templates from similar documents

Using very basic techniques (zone segmentation + OPTICS) I was able to organize a set of around 10^4 business documents (invoices, receipts) into hierarchy of clusters of documents of similar layout. ...
dzieciou's user avatar
  • 697
2 votes
2 answers
361 views

How to implement Semantic Search in R or Python

I have a task to provide semantic searching capabilities. For example, if I have a dataset of resume and if I search for "machine learning" than it should return me all resumes which have data science-...
Yash Kanojia's user avatar
1 vote
3 answers
3k views

Fastest way for 1 vs all lookup on embeddings

I have a dataset with about 1 000 000 texts where I have computed their sentence embeddings with a language model and stored them in a numpy array. I wish to compare a new unseen text to all the 1 ...
Isbister's user avatar
  • 183
1 vote
2 answers
394 views

Dynamic clustering for text documents

I have few hundred thousands of text documents. Some of them are pretty similar - they differ just in ex. names or some numbers, all other text is the same. I would like to cluster these documents, so ...
MaticDiba's user avatar
  • 651
1 vote
1 answer
5k views

Text Similarities: which nlp methods to use?

I have data where there is text for each user A visiting a business B. I want to find similarity between each user using their text. Question 1: Which NLP method should I start with? I have tried ...
jdwins11's user avatar
1 vote
1 answer
96 views

document similarity using LDA probabilities

Let us say I have a LDA model trained on a corpus of text. I would like to know, for a newly given document, which one from the corpus is closet to it. But, to do so, I want to use probabilities ...
aRedDish's user avatar
1 vote
1 answer
99 views

Deep learning techniques for concept similarity?

Given a corpus of product descriptions (say, vacuum cleaners), I'm looking for a way to group the documents that are all of the same type (where a type can be ...
rodrigo-silveira's user avatar
1 vote
1 answer
810 views

Building a graph out of a large text corpus

I'm given a large amount of documents upon which I should perform various kinds of analysis. Since the documents are to be used as a foundation of a final product, I thought about building a graph out ...
kevin_was_here's user avatar
1 vote
1 answer
58 views

Document Similarity with User Preference

To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
JoyfulPanda's user avatar
1 vote
1 answer
135 views

Document ranking on a web scraped dataset without any labelled data

I want to create a document ranking model which returns similar rows in the dataset for a sample query. The text in this corpus is standard english but without any labels (ie no query-related ...
sarva's user avatar
  • 11
1 vote
1 answer
65 views

Evaluation of recommendation systems

I have developed a content-based recommendation system and it is working fine. The input is a set of documents={d1,d2,d3,...,dn} and the output will be Top N similar documents for a given document ...
Raj's user avatar
  • 159
1 vote
1 answer
394 views

What is considered short and long text in NLP (document similarity)

What is considered short and long text in NLP? I'm working on a dataset that contains documents from 10 to 600 words and I'm asking myself if I should treat them differently. Also, I haven't found a ...
jonas's user avatar
  • 143
1 vote
1 answer
519 views

pandas.isna() vs pandas.DataFrame.isna()

I've seen the two documentation pages for pandas.isna() and pandas.DataFrame.isna() but the difference is still unclear to me. Could someone explain the difference to me using examples?
Mountain Scott's user avatar
1 vote
1 answer
95 views

Checking TF-IDF Results

I am working with TF-IDF and cosine similarity to do document comparisons and given a document, which document in the data is the most similar. However, sometimes it returns a high similarity between ...
user1111111111's user avatar
1 vote
1 answer
8k views

Cosine similarity between query and document confusion

I am going through the Manning book for Information retrieval. Currently I am at the part about cosine similarity. One thing is not clear for me. Let's say that I have the tf idf vectors for the ...
AutisticRat's user avatar
1 vote
2 answers
1k views

What techniques should I use to compare the similarity between a bunch of texts?

If I have a list of job postings stored as raw texts and I want to compare the similarity of all the job postings to a given resume, what techniques or algorithms should I use? I'm thinking of ...
Steven Chen's user avatar
1 vote
1 answer
1k views

Classification of scanned documents in pdf files using deep learning or NLP

I know classifying images using cnn but I have a problem where I have multiple types of scanned documents in a pdf file on different pages. Some types of scanned documents present in multiple pages ...
Sherlock 's user avatar
1 vote
0 answers
172 views

Performing actual deduplication using LSH

I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
Marcin Zablocki's user avatar
1 vote
0 answers
175 views

Matching documents from different sets with tfidf and cosine distance

I have two different set of documents S1, S2, with 30 text documents each. Using some text representation method, such as tfidf ...
lynx's user avatar
  • 113
1 vote
2 answers
215 views

Evaluate document similarity / content-based recommender system

I'm planning on building a basic content-based recommender system with word2vec and cosine similarity. The data consists of 300k documents in varying length. How do I evaluate my model if I have no ...
jonas's user avatar
  • 143
1 vote
0 answers
34 views

Document Similarity to List of Words in Sentiment Analysis [closed]

How would you go about finding document similarity to a list of words in Sentiment Analysis? Looking find document similarity to multiple lists of words in sentiment analysis. I had been working on ...
JohnT's user avatar
  • 111
1 vote
0 answers
1k views

How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?

I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data. At present I ...
Vikrant's user avatar
  • 11
1 vote
0 answers
36 views

What is the best technique to transform documents into vectors?

What is the best algorithm between doc2vec and Singular Value Decomposition (SVD) to transform a set of 600 documents of around 1000 words each into vectors ?
Alexis Pister's user avatar
1 vote
0 answers
630 views

Doc2vec most similar document to a query string

I'm working on a project and I created doc2vec representation of different academics which include their patents and publications etc. For each publication and patent I have information such as title ...
qiqi's user avatar
  • 11
1 vote
0 answers
133 views

Counting similarity between two GIT commits

I am writing as part of my master thesis work an analysis of developers' work. I am looking for a new parameters to be consumed in Topological Data Analysis process. I would like to compare a ...
MKMZ's user avatar
  • 11
1 vote
0 answers
107 views

What do I️ do with an array of log probabilities? Doc2vec

I️ am working on a document classifier utilizing the Gensim libabry (doc2vec). After creating the model, I️ run the doc2vec.score() function on a document collection per document. According to the ...
ldavila07's user avatar
0 votes
1 answer
608 views

Finding similar articles in realtime [closed]

I want to build a large document (news article) searchable database, such as when adding a new article I will be able to quickly find X most similar articles from it. What is the right tech/algorithm/...
rubmz's user avatar
  • 103
0 votes
1 answer
389 views

Cosine vs Manhattan for Text Similarity [closed]

I'm storing sentences in Elasticsearch as dense_vector field and used BERT for the embedding so each vector is 768 dim. ...
Mohy Mohamed's user avatar
0 votes
1 answer
67 views

How does RAG query affect the similarity search?

I have a RAG pipeline where I want to extract a piece of information called "X" In a regular RAG pipeline, there is a query entered by the user. Then, ...
ahmedmoh123's user avatar
0 votes
1 answer
39 views

How to filter Items in Recommender Systems?

I have a Recommender System which recommends Articles based on Similarity from 3 Features, "Page-Title, Article Content, Tags". But some of the Articles are NSFW(Related to Adult Topics). I ...
m2rik's user avatar
  • 321