3
$\begingroup$

I'm doing an online course and one of the assignment says:

To avoid parameters from exploding or becoming highly correlated, it is helpful to augment our cost function with a Gaussian prior: this tends to push parameter weights closer to zero, without constraining their direction, and often leads to classifiers with better generalization ability.

Exactly what does "augment our cost function with a Gaussian prior" mean, and how could I go about doing that?

I tried searching on google but didn't find anything helpful.

$\endgroup$
1
$\begingroup$

Let's look at the example of linear regression. Instead of deriving it from solving the normal equations, we can motivate it by thinking about it as finding the conditional distribution $P(y|x)$. Let's assume that this distribution follows a Gaussian distribution with fixed variance $\sigma^2$ and mean $\hat{y}(x,w)$ and weights $w$. Assuming that samples are i.i.d. we can easily show that applying maximum-likelihood gives the same values of $w$ than minimizing the mean squared error. See example 5.5.1 for a mathematical explanation.

If we additionally assume a prior Gaussian distribution for $P(w)$ (w.l.o.g. we can assume unit variance $I$), we can now show that the posterior $P(w | x,y)$ of $w$ is also Gaussian distributed (that is because the prior is "conjugate") with variance $\frac{1}{\alpha} I$ and we recover linear regression with weight decay $\alpha w^T w=\alpha \lVert w \rVert^2_2$ (i.e. $L^2$ or Tikhonov regularization).

Intuitively, a Gaussian distribution's probability mass is centered around the mean (assumed here to be zero). If we assume such a distribution for the coefficients in the linear regression (as our prior knowledge), linear regression will favor coefficients close to zero.

All of this is nicely explained here (chapter 5) or here (chapter 18).

| improve this answer | |
$\endgroup$
-1
$\begingroup$

It was hard to look for anything based on the information. Still, I found this interesting paper- Augmented Functional Time Series Representation and Forecasting with Gaussian Processes on the web.

Just explore it. Hope it will help you upto some extent. Cheers!

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.