Where can we find the application of bayes's theorem in Bayesian optimiation with gaussian processing

Question

I am trying to learn bayesian optimisation by following this tutorial. However, until now I don't get the relation between bayes's theorem to the gaussian process formalism.

Any ideas?

Answer 1

It is a 49 page long paper, so following observations are based only on a cursory reading.

1. The optimisation is for finding best value of parameters for cost function of machine learning models.
2. Rather than finding a fixed value of the parameters, it is assumed that the parameters come from statistical distribution and the task is to find the nature/shape of this statistical distribution.
3. Bayes theorem tells you that if you have prior beliefs and evidence (data), you can go to posterior.
4. They start with the assumption that prior distribution of the parameters is Gaussian. The task is to then find the posterior.
5. Since there are multiple parameters and not single variable, Gaussian process comes into picture rather than single Gaussian distributed random variable.
6. Bayes theorem comes into picture while going from priors to posteriors. The optimisation is solved using sampling technique like Monte Carlo sampling. Reading up about MCMC sampling method will help you see the connection.