Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I would like to compare two methods: locality sensitivity hashing and document embedding to get the similarity between two documents. Both of those methods encode information of a document in a vector which I would like to use to find similar documents in a very large corpus (potentially more than 100 000 documents). Have anybody ever compared those two methods and what are the advantages of each of them.
Cheers in advance
Locality sensitive hashing (LSH) is a search technique. With it, similar documents get the same hash with higher probability than dissimilar documents do. LSH is designed to allow you to build lookup tables to efficiently search large data sets for items similar to a given item. It is also a probabilistic method in that it can generate false positives and false negatives. While there are ways to train LSH, most LSH is untrained. That's because LSH has been studied more in the search setting than the machine learning setting.
Embeddings are a machine learning technique to capture semantic information for use in some downstream task, such as clustering or classification. Typically semantically similar items get similar (but not the same) embeddings. Embeddings are trained from data. There are many unsupervised algorithms (word2vec, glove) and there are supervised methods too (auto-encoders, hidden layer output from deep models).
Embeddings can be used to map items into a space where near neighbor search would find semantically similar items. However, on large data sets you would still need to index the data to search efficiently, which raises the possibility of doing LSH on embeddings. That way you get the benefit of a trained model that learns the distribution of your data set and the benefit of a fast lookup table.
