I'm trying to get grasp of Bayes regularization algorithm. List of symbols 1st:
$F$ - objective function
$\gamma$ - regularization parameter
$M$ - number od neural network weights
$N$ - number of data tuples
$e$ - modeling error
$w$ - net weight
$D$ - data set input-output pair
$H$ - Hess matrix of objective function computed based on Jacoby's matrix

I get it that minimalising an objective function: $$ F = \gamma\sum\limits _{j=1} ^Mw_j^2 + (1-\gamma)\sum\limits_{i=1}^Ne_i^2 $$ is equivalent to maximising likelihood $P(w|D,\gamma)$, which can be computed according to Bayes' theorem: $$ P(w|D,\gamma) = \frac{P(D|w,\gamma)P(w|\gamma)}{P(D|\gamma)} $$ Likelihood $P(w|\gamma)$ is assumed to be a gaussian one and can be computed as: $$ P(w|\gamma)=\left(\frac{\gamma}{2N}\right)^{\frac{M}{2}}\cdot e^{-\frac{\gamma}{2}w^Tw} $$ Likelihood $P(D|\gamma)$ can be computed too as: $$ P(D|\gamma)=\left(\frac{\pi}{\gamma}\right)^{-\frac{N}{2}}\left(\frac{\pi}{1-\gamma}\right)^{-\frac{M}{2}}\frac{(2\pi)^\frac{M}{2}\cdot -e^{-F(w)}}{\sqrt{|H|}} $$ And now my questions are: how is $P(D|w,\gamma)$ computed? Or maybe is there any special assumptions about it? And should I minimalise or maximise $P(w|D,\gamma)$ to minimalise objective function $F$.
I base on those articles: Article1 Article2

If anyone is able to explain those thing to me, I'd be most grateful.
Thank you,


I have found answer for my question here.

If anyone needs it in future: all above likelihoods are assumed to be a gaussian distributions. Likelihood $P(\gamma)$ is assumed to be uniform. In the article it is shown how exactly are they defined and how to compute further is described in detail in article linked above.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.