2
$\begingroup$

I would like to compare two methods: locality sensitivity hashing and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a vector which I would like to use to find similar documents in a very large corpus (potentially more than 100 000 documents). Have anybody ever compared those two methods and what are the advantages of each of them.

Cheers in advance

$\endgroup$
1
  • $\begingroup$ Thanks for your answer, so if I were to identify documents which are related, I'd rather use document embedding than locality sensitivity hashing? $\endgroup$ Sep 27, 2019 at 10:28

1 Answer 1

1
$\begingroup$

Locality sensitive hashing (LSH) is a search technique. With it, similar documents get the same hash with higher probability than dissimilar documents do. LSH is designed to allow you to build lookup tables to efficiently search large data sets for items similar to a given item. It is also a probabilistic method in that it can generate false positives and false negatives. While there are ways to train LSH, most LSH is untrained. That's because LSH has been studied more in the search setting than the machine learning setting.

Embeddings are a machine learning technique to capture semantic information for use in some downstream task, such as clustering or classification. Typically semantically similar items get similar (but not the same) embeddings. Embeddings are trained from data. There are many unsupervised algorithms (word2vec, glove) and there are supervised methods too (auto-encoders, hidden layer output from deep models).

Embeddings can be used to map items into a space where near neighbor search would find semantically similar items. However, on large data sets you would still need to index the data to search efficiently, which raises the possibility of doing LSH on embeddings. That way you get the benefit of a trained model that learns the distribution of your data set and the benefit of a fast lookup table.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.