Linked Questions

15 votes
4 answers
14k views

Alternatives to TF-IDF and Cosine Similarity when comparing documents of differing formats

I've been working on a small, personal project which takes a user's job skills and suggests the most ideal career for them based on those skills. I use a database of job listings to achieve this. At ...
Richard Knoche's user avatar
6 votes
3 answers
6k views

Weighted sum of word vectors for document similarity

I have trained a word2vec model on a corpus of documents. I then compute the term frequency (the same Tf in TfIDF) of each word in each document, multiply each words Tf by its corresponding word ...
PyRsquared's user avatar
  • 1,604
4 votes
3 answers
12k views

How to measure the similarity between two text documents?

Assume, I have 100 text documents, and I want to cluster those documents. The first step is the construct pairwise similarity matrix 100X100 for the documents My ...
jason's user avatar
  • 329
4 votes
2 answers
3k views

Document similarity: Vector embedding versus BoW performance?

I have a collection of documents, where each document is rapidly growing with time. The task is to find similar documents at any fixed time. I have two potential approaches: A vector embedding (...
Alec Matusis's user avatar
3 votes
2 answers
2k views

Cluster documents based on topic similarity

I have set of documents where I have assigned topics per each document. E.g., Topics of document 1 -> 1.0 Science, 1.0 politics, 0.8 History, 0. 8 Information and Technology Now I want to cluster ...
Smith's user avatar
  • 529
2 votes
2 answers
351 views

Document similarity

I have close to 50000 documents in plain text format. Is there a way in which I can group similar documents together? Similarity mostly here is the content similarity. Will transforming the text ...
praneeth's user avatar
  • 149
1 vote
1 answer
2k views

Use embeddings to find similarity between documents

I need to find cosine similarity between two text documents. I need embeddings that reflect order of the word sequence, so I don't plan to use document vectors built with bag of words or TF/IDF. ...
dokondr's user avatar
  • 295
1 vote
2 answers
462 views

Word2Vec - document similarity

Lets say I have text data for different documents from 2005 - 2015. I want to compare the similarity between $t$ and $t-1$ documents. So I take the document at 2006 and compare it with the document at ...
user8959427's user avatar
1 vote
1 answer
366 views

What is considered short and long text in NLP (document similarity)

What is considered short and long text in NLP? I'm working on a dataset that contains documents from 10 to 600 words and I'm asking myself if I should treat them differently. Also, I haven't found a ...
jonas's user avatar
  • 143
1 vote
1 answer
7k views

Cosine similarity between query and document confusion

I am going through the Manning book for Information retrieval. Currently I am at the part about cosine similarity. One thing is not clear for me. Let's say that I have the tf idf vectors for the ...
AutisticRat's user avatar
1 vote
2 answers
212 views

Evaluate document similarity / content-based recommender system

I'm planning on building a basic content-based recommender system with word2vec and cosine similarity. The data consists of 300k documents in varying length. How do I evaluate my model if I have no ...
jonas's user avatar
  • 143