Questions tagged [similar-documents]
The similar-documents tag has no usage guidance.
61
questions
10
votes
1
answer
688
views
What is the difference between fasttext and DANs in document classification?
I came across two interesting papers that describe promising approaches for document classification using word embedding.
1. The fasttext algorithm
Described in the paper Bag of Tricks for ...
- 201
8
votes
2
answers
5k
views
Text similarity with sentence embeddings
I'm trying to calculate similarity between texts with various lengths. My current approach is following:
Using Universal Sentence Encoder, I convert text to a set of vectors.
I average these vectors ...
6
votes
1
answer
1k
views
Can I use euclidean distance for Latent Dirichlet Allocation document similarity?
I have a Latent Dirichlet Allocation (LDA) model with $K$ topics trained on a corpus with $M$ documents. Due to my hyper parameter configurations, the output topic distributions for each document is ...
- 1,564
6
votes
1
answer
188
views
How to compute document similarities in case of source codes?
I try to detect the probability of common authorship (person, company) of different kind of source code texts (webpages, program codes). My first idea is to apply the usual NLP tools like any token ...
- 8,247
5
votes
1
answer
818
views
TS-SS and Cosine similarity among text documents using TF-IDF in Python
A common way of calculating the cosine similarity between text based documents is to calculate tf-idf and then calculating the linear kernel of the tf-idf matrix.
TF-IDF matrix is calculated using ...
- 153
5
votes
1
answer
578
views
Using Spark for finding similar users to a user?
I read about
https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html
but couldn't find a spark library for this implementation.
I have columnar string dataset.
I have a dataset ...
- 191
4
votes
3
answers
11k
views
How to measure the similarity between two text documents?
Assume, I have 100 text documents, and I want to cluster those documents.
The first step is the construct pairwise similarity matrix 100X100 for the documents
My ...
- 309
4
votes
2
answers
169
views
Automatic code checking
I have some experience in machine learning, mainly clustering and classifiers. However, I am somewhat of a newbie when it comes to NLP.
That said I am aware of all the various issues and ...
- 41
4
votes
2
answers
495
views
Data wrangling for a big set of docx files advice!
I'm looking for some advice on a data wrangling problem I'm trying to solve. I've spent a week solid taking different approaches and nothing seems to be quite perfect. Just FYI, this is my first big (...
- 41
3
votes
1
answer
3k
views
Training Doc2Vec and Word2Vec at the same time
As far as I can tell the typical Doc2Vec implementation (e.g. Gensim) first trains the word vectors and afterwards the document vectors were the word vectors are fixed.
If my goal is that ...
- 31
3
votes
3
answers
2k
views
Which algorithm Doc2Vec uses?
Like Word2vec is not a single algorithm but combination of two, namely, CBOW and Skip-Gram model; is Doc2Vec also a combination of any such algorithms? Or is it an algorithm in itself?
- 289
3
votes
2
answers
73
views
Gensim doc2vec error: KeyError: "word 'senseless' not in vocabulary"
I am new to machine learning and tried doc2vec on quora duplicate dataset. new_dfx has columns 'question1' and 'question2' which has preprocessed questions in each row. Following is the tagged ...
2
votes
2
answers
353
views
ways to represent document by its keyword vectors
I have documents, say for example, D1, D2, D3... Dm.
Every Di has its individual components or keywords k1, k2, k3,... kn, where ki is an n-dimensional vector. The number of individual components ...
- 285
2
votes
2
answers
324
views
Document similarity
I have close to 50000 documents in plain text format.
Is there a way in which I can group similar documents together? Similarity mostly here is the content similarity.
Will transforming the text ...
- 149
2
votes
2
answers
768
views
Hash function for text documents that maps similar documents to the same value
I have a web site that process text documents (typically 10-100 pages) submitted by users. Each time a user submits a document, I'd like to store a hash of the document, but I'd like similar ...
- 121
2
votes
1
answer
1k
views
Document embedding vs locality sensitive hashing for document clustering
I would like to compare two methods: locality sensitivity hashing and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a ...
- 55
2
votes
1
answer
143
views
How to classify a document by image? [closed]
I need an opens source solution to classify a document. I do not want to use NLP i need only to check the look and feel.
I tried OpenCV. I have a template and i need to match it.
...
- 23
2
votes
0
answers
48
views
Preprocessing for Document Similarity Using Doc2Vec
I'm trying to determine document similarity using Doc2Vec on a large series of legal opinions, which can contain some highly jargonistic language and phrases (e.g. en banc, de novo, etc.). I'm ...
- 21
2
votes
0
answers
391
views
Unsupervised document similarity state of the art
I have a set of N documents with lengths ranging from 0 to more than 20000 characters. I want to calculate a similarity score between 0 and 1 between all pairs of documents where a higher number ...
- 21
2
votes
2
answers
136
views
Semantic Search
There is a problem we are trying to solve where we want to do semantic search on our set of data,
i.e we have a domain specific data (example: sentences talking about automobiles)
Our data is just a ...
2
votes
1
answer
265
views
How to implement Semantic Search in R or Python
I have a task to provide semantic searching capabilities. For example, if I have a dataset of resume and if I search for "machine learning" than it should return me all resumes which have data science-...
- 21
1
vote
3
answers
2k
views
Fastest way for 1 vs all lookup on embeddings
I have a dataset with about 1 000 000 texts where I have computed their sentence embeddings with a language model and stored them in a numpy array.
I wish to compare a new unseen text to all the 1 ...
- 173
1
vote
2
answers
378
views
Dynamic clustering for text documents
I have few hundred thousands of text documents. Some of them are pretty similar - they differ just in ex. names or some numbers, all other text is the same. I would like to cluster these documents, so ...
- 641
1
vote
1
answer
5k
views
Text Similarities: which nlp methods to use?
I have data where there is text for each user A visiting a business B. I want to find similarity between each user using their text.
Question 1: Which NLP method should I start with?
I have tried ...
- 23
1
vote
1
answer
50
views
document similarity using LDA probabilities
Let us say I have a LDA model trained on a corpus of text.
I would like to know, for a newly given document, which one from the corpus is closet to it.
But, to do so, I want to use probabilities ...
- 31
1
vote
1
answer
36
views
Deep learning techniques for concept similarity?
Given a corpus of product descriptions (say, vacuum cleaners), I'm looking for a way to group the documents that are all of the same type (where a type can be ...
- 150
1
vote
1
answer
310
views
Building a graph out of a large text corpus
I'm given a large amount of documents upon which I should perform various kinds of analysis. Since the documents are to be used as a foundation of a final product, I thought about building a graph out ...
1
vote
1
answer
33
views
Document Similarity with User Preference
To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
- 111
1
vote
1
answer
40
views
Evaluation of recommendation systems
I have developed a content-based recommendation system and it is working fine. The input is a set of documents={d1,d2,d3,...,dn} and the output will be Top N similar documents for a given document ...
- 159
1
vote
1
answer
245
views
What is considered short and long text in NLP (document similarity)
What is considered short and long text in NLP?
I'm working on a dataset that contains documents from 10 to 600 words and I'm asking myself if I should treat them differently. Also, I haven't found a ...
- 143
1
vote
1
answer
443
views
pandas.isna() vs pandas.DataFrame.isna()
I've seen the two documentation pages for pandas.isna() and pandas.DataFrame.isna() but the difference is still unclear to me. Could someone explain the difference to me using examples?
1
vote
1
answer
75
views
Checking TF-IDF Results
I am working with TF-IDF and cosine similarity to do document comparisons and given a document, which document in the data is the most similar. However, sometimes it returns a high similarity between ...
1
vote
1
answer
7k
views
Cosine similarity between query and document confusion
I am going through the Manning book for Information retrieval. Currently I am at the part about cosine similarity. One thing is not clear for me.
Let's say that I have the tf idf vectors for the ...
- 123
1
vote
2
answers
1k
views
What techniques should I use to compare the similarity between a bunch of texts?
If I have a list of job postings stored as raw texts and I want to compare the similarity of all the job postings to a given resume, what techniques or algorithms should I use?
I'm thinking of ...
- 152
1
vote
1
answer
826
views
Classification of scanned documents in pdf files using deep learning or NLP
I know classifying images using cnn but I have a problem where I have multiple types of scanned documents in a pdf file on different pages. Some types of scanned documents present in multiple pages ...
- 11
1
vote
1
answer
86
views
Document ranking on a web scraped dataset without any labelled data
I want to create a document ranking model which returns similar rows in the dataset for a sample query. The text in this corpus is standard english but without any labels (ie no query-related ...
- 11
1
vote
0
answers
144
views
Performing actual deduplication using LSH
I have a huge dataset (>10M) of text files, which I try to de-duplicate - not only in terms of trivial duplicates, but also "near-duplicates", given some similarity threshold. I know that ...
- 121
1
vote
0
answers
116
views
Matching documents from different sets with tfidf and cosine distance
I have two different set of documents S1, S2, with 30 text documents each.
Using some text representation method, such as tfidf ...
- 113
1
vote
2
answers
172
views
Evaluate document similarity / content-based recommender system
I'm planning on building a basic content-based recommender system with word2vec and cosine similarity.
The data consists of 300k documents in varying length.
How do I evaluate my model if I have no ...
- 143
1
vote
0
answers
32
views
Document Similarity to List of Words in Sentiment Analysis [closed]
How would you go about finding document similarity to a list of words in Sentiment Analysis?
Looking find document similarity to multiple lists of words in sentiment analysis. I had been working on ...
- 111
1
vote
0
answers
45
views
Extracting document templates from similar documents
Using very basic techniques (zone segmentation + OPTICS) I was able to organize a set of around 10^4 business documents (invoices, receipts) into hierarchy of clusters of documents of similar layout. ...
- 687
1
vote
0
answers
1k
views
How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?
I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data.
At present I ...
- 11
1
vote
0
answers
31
views
What is the best technique to transform documents into vectors?
What is the best algorithm between doc2vec and Singular Value Decomposition (SVD) to transform a set of 600 documents of around 1000 words each into vectors ?
- 567
1
vote
0
answers
589
views
Doc2vec most similar document to a query string
I'm working on a project and I created doc2vec representation of different academics which include their patents and publications etc. For each publication and patent I have information such as title ...
- 11
1
vote
0
answers
119
views
Counting similarity between two GIT commits
I am writing as part of my master thesis work an analysis of developers' work. I am looking for a new parameters to be consumed in Topological Data Analysis process. I would like to compare a ...
- 11
1
vote
0
answers
103
views
What do I️ do with an array of log probabilities? Doc2vec
I️ am working on a document classifier utilizing the Gensim libabry (doc2vec). After creating the model, I️ run the doc2vec.score() function on a document collection per document.
According to the ...
- 11
0
votes
1
answer
245
views
Cosine vs Manhattan for Text Similarity [closed]
I'm storing sentences in Elasticsearch as dense_vector field and used BERT for the embedding so each vector is 768 dim. ...
- 45
0
votes
1
answer
597
views
Finding similar articles in realtime [closed]
I want to build a large document (news article) searchable database, such as when adding a new article I will be able to quickly find X most similar articles from it.
What is the right tech/algorithm/...
- 103
0
votes
1
answer
33
views
How to filter Items in Recommender Systems?
I have a Recommender System which recommends Articles based on Similarity from 3 Features, "Page-Title, Article Content, Tags". But some of the Articles are NSFW(Related to Adult Topics). I ...
- 321
0
votes
1
answer
34
views
How to implement a basic query management and recommendation system
I'm trying to prototype a system where given a textual query (e.g. a question), I get a list of most relevant documents/questions among a pool of available documents/questions (similar to what we see ...
- 197