Approaches to Bag-Of-Words Information Retrieval

Question

I'm interested in an overview of the modern/state-of-the-art approaches to bag-of-words information retrieval, where you have a single query $q$ and a set of documents which you hope to rank by relevance $d_1,...,d_n$.

I'm specifically interested in approaches which require absolutely no linguistic knowledge and rely on no outside linguistic or lexical resources for boosting performance (such as thesauri or prebuilt word-nets and the like). Thus where a ranking is produced entirely by evaluating query-document similarity and where the problems of synonymy and polysemy are overcome by exploiting inter-document word co-occurence.

I've spent some amount of time in the literature (reading papers/tutorials), however there is so much information out there it's hard to get a bird's eye view. The best I can make out is that modern approaches involve some combination of a weighted vector space model (such as a generalized vector space model, LSI, or a topic-based vector space model using LDA), in conjunction with pseudo-relevance feedback (using either Rocchio or some more advanced approach).

All these vector space models tend to use cosine-similarity as the similarity function, however I have seen some literature discussing similarity functions which are more exotic.

Is this basically where we currently are in attacking this particular type of problem?

Answer 1

While reading blogs and papers is helpful to identify the latest and greatest, having a solid foundation helps a lot, too. But I assume you already have gone over Manning's great (and free in e-book form) book on IR, right?

http://nlp.stanford.edu/IR-book/

It contains information on creating your own thesaurus from your document collection to solve synonymy problems, LSA for polysemy, etc..

As for similarity measures, you will see there that Okapi BM25 (Robertson et al.) is considered superior to cosine similarity (but more expensive to implement and run). Regarding the current state of the art, there was a small emergence of Bayesian Network-based classifiers in the early nineties (starting with Turtle & Croft), but that went quiet for a while. However, right now, using BNs for IR is again finding some revival, particularly in biomedical IR. In that respect, I think most ongoing work is directed towards using Bayesian models incl. topic models and deep learning for word-sense disambiguation (WSD) and semantic similarity. Here is a pointer to a recent paper with good references on the topic.

http://arxiv.org/abs/1412.6629

Answer 2

The similarity function at the core of the method will define all the values for your distances $d_1,d_2, \ldots, d_n$. The initial query should have some words as a reference point to compare to the words in the document. Not knowing whether the query is a sentence or arbitrary list, you are restricted to a method that does some kind of histogram comparison of the frequency of the words matching in the documents. You can perform naive summations of keyword mappings counts, look at keyword likelihoods in the normalized distributions, or give a distribution of weighting based on the strongest matches. More exotic functions will be based on your prior belief of how the words should be compared. Working within a Bayesian Framework you can see your prior assumptions explicitly. Cosine similarity or any other vector based measure will be slightly arbitrary without knowing the desired nature of comparison between query and document.

There is not much more you can do without looking at some type of features, or attempt to cross compare the documents together, or use the initial query's structure. In short, my answer is to use normalized frequency similarities of the document to the queries and produce a ranking, and with more specific goals in mind to apply measures like cosine similarity on test datasets to search for the best measure.