How does bayesian optimization with gaussian processes work?

Question

Could someone explain in simple words

1. what are gaussian processes
2. how does bayesian optimization work
3. and their combination?

Answer 1

Gaussian processes are a way of interpolating between data using a kernel fit, but with a covariance matrix. Think of it as kernel regression, but with confidence intervals that can be drawn at every point.

This diagram from this medium article is explains how a combination of a Gaussian process and an acquisition function works in Bayesian optimization:

An optimization problem is one that has an objective, for example, you might want to find a global minimum. Given the predictions and the confidence interval of a Gaussian process, Bayesian optimization uses an acquisition function to choose observations that would advance that objective. The acquisition function is a combination of that objective, and a balance between exploration and exploitation.

The key concepts you should grasp are:

1. Bayesian optimization balances between exploring new and uninformed areas without data, and exploiting known information from pre-existing data.
2. This continually improves a Gaussian process model, so that it makes better decisions about what to observe next.
3. All of this is to optimize for a particular objective