State of the art approaches for Information retrieval tasks based on deep learning

Question

I am interested in the state of the art approaches for information retrieval (IR) tasks, where you have a single query and a set of documents and the IR model will give you the best matched document.

I have worked on vector space models (tfidf-cosine similarirty ) and LSA.

I have also tried Wordnet, NER, fuzzy matching etc for improving the accuracy.

Now I would like to know how to improve accuracy of IR tasks by applying nueral networks, word embeddingstopic models etc by capturing more context/sematics information

Answer 1

Deep learning extends the ability of information retrieval (IR) systems.

Deep learning has proved to be a powerful tool for feature engineering. This improvement in feature engineering, over manual construction, has improved the quality of any machine learning system, including information retrieval.

Specifically, word embeddings create a dense, vectorized representation of words that encodes the semantic relationships based on co-occurrence. Both documents and queries can be projected into the same latent space and nearest-neighbors can be found. Word embeddings can be used to extend the posting index to by finding related terms. They can also be used for query-expansion / query rewriting.