Query similarity: how much data is used in practice?

Question

I recently read Similarity Measures for Short Segments of Text (Metzler et al.). It describes basic methods for measuring query similarity, and in the paper, the data consists of queries and their top results. Results are lists of page urls, page titles, and short page snippets. In the paper, the authors collect 200 results per query.

When using the public Google APIs to retrieve results, I was only able to collect 4-10 results per query. There's a substantial difference between 10 and 200. Hence, how much data is commonly used in practice to measure query similarity (e.g., how many results per query)?

References are a plus!

Answer 1

When using the public Google APIs to retrieve results, I was only able to collect 4-10 results per query.

Here's how to get more than 10 results per query: https://support.google.com/customsearch/answer/1361951?hl=en

Google Custom Search and Google Site Search return up to 10 results per query. If you want to display more than 10 results to the user, you can issue multiple requests (using the start=0, start=11 ... parameters) and display the results on a single page. In this case, Google will consider each request as a separate query, and if you are using Google Site Search, each query will count towards your limit.

There are other search engine APIs as well (e.g., Bing)