How does Google categorize results from its image search?

Question

While doing a Google image search, the page displays some figured out categories for the images of the topic being searched for. I'm interested in learning how this works, and how it chooses and creates categories.

Unfortunately, I couldn't find much about it at all. Is anyone able to shed some light on algorithms they may be using to do this, and what basis these categories are created from?

For example, if I search for "animals" I get the categories:

"cute", "baby", "wild", "farm", "zoo", "clipart".

If I go into "wild", I then have subcategories:

"forest", "baby", "africa", "clipart", "rainforest", "domestic".

Answer 1

I am not working at Google, but I think it is some sort of recommendation system based on the words which millions of users searched before. So those people who search for "animals" often search for "wild animals" for example. Like in many online stores they recommend you to buy something in addition to the product you are looking for based on the previous purchases of other users.

There are many approaches how to build such recommendation system using machine learning, no one knows for sure what google uses.

Answer 2

I thought of expanding a bit on the answer by Stanpol. While recommendation system is one approach of suggesting related queries, one more standard information retrieval based approach is the query expansion technique.

Generally speaking, query expansion involves selecting additional terms from the top ranked documents retrieved in response to an initial query. Terms are typically selected by a combination of a term scoring function such as tf-idf and a co-occurrence based measure.

For example, in response to a query term "animal", a term selection function may choose the term "zoo", because

• "zoo" may be a dominating term (high tf-idf) in the top (say 10) documents retrieved in response to the query "animal"
• "zoo" may co-occur frequently (in close proximity) with the original query term "animal" in these documents

Answer 3

Google isn't going to give away their proprietary work, but we can speculate.

Here's what I can gather from my limited usage:

1. The recommendations do not seem to be user, geography, or history specific.
2. There is never an empty recommendation (one that returns no results)
3. There is not always a recommendation (some searches just return images)
4. The recommendations are not always the same (consecutive searches sometimes return different recommendations)
5. Result ordering shifts regularly (search for a specific image and it won't always be in the same place)
6. Very popular searches seem to be pre-calculated and more static than unpopular searches.
7. Recommendations are not always one additional word, and recommendations do not always include the base query.

It seems to me that they do this based on the history of the general end user population, they rotate recommendations when there are many popular ones, and they do some additional processing to determine that the result set is of a reasonable size.

I would postulate that it works as follows:

1. Use consecutive search strings from users (short-to-long-tail searches) as training data for a machine-learning algorithm.
2. Run searches that occur > N amount of times a week against that recommendation algorithm.
3. Validate and clean results.
4. Push them out to the general population in rotation/A-B testing.
5. Track click-throughs.
6. Refine results over time.

Answer 4

I am not working at Google, but According to this blog post "Graph-powered Machine Learning at Google" from Oct 2016, they use an in-house software called "Google Expander", and "in conjuction with neural networks" it performs semisupervised learning based on graph-theory.

The blog post says, this arXiv paper, Large Scale Distributed Semi-Supervised Learning Using Streaming Approximation from 2015 gives a longer technical answer to the question.

2016 arXiv paper, on smart gmail response