0
$\begingroup$

This is a really simple neural network with backprop. If one had to apply bayesian "inferences" to update the weights and biases, what would change in the code.

#Forward Propogation
hidden_layer_input1=np.dot(X,wh)
hidden_layer_input=hidden_layer_input1 + bh               # linear transformation
hiddenlayer_activations = sigmoid(hidden_layer_input)     # non-linear transformation
output_layer_input1=np.dot(hiddenlayer_activations,wout)
output_layer_input= output_layer_input1+ bout             # linear transformation
output = sigmoid(output_layer_input)

#Backpropagation
E = y-output
slope_output_layer = derivatives_sigmoid(output)
slope_hidden_layer = derivatives_sigmoid(hiddenlayer_activations)
derivative_output = E * slope_output_layer
Error_at_hidden_layer = derivative_output.dot(wout.T)
derivative_hiddenlayer = Error_at_hidden_layer * slope_hidden_layer
wout += hiddenlayer_activations.T.dot(d_output) *lr
bout += np.sum(derivative_output, axis=0,keepdims=True) *lr
wh += X.T.dot(d_hiddenlayer) *lr                          # update weight
bh += np.sum(d_hiddenlayer, axis=0,keepdims=True) *lr     # update bias

If my understanding is correct of using bayes method to derive the weights and biases, it would help in getting to those values faster, leaving behind unnecessary loops for weights which would not converge.

Apart from that, coding and observing it is the best bet to get a real perception.

Would like an understanding.

$\endgroup$
2
  • $\begingroup$ It's not that simple, and I don't know what you mean by "coding and observing it is the best bet to get a real perception". If you want to go Bayesian you have two options: variational- and Monte Carlo methods. Are you familiar with them? I suggest implementing Bayesian linear regression before trying a neural network. $\endgroup$
    – Emre
    Jul 13, 2017 at 19:55
  • $\begingroup$ I think this question is incomprehensible. For starters, there are no loops in the code, so I don't understand why the user asks for "leaving behind unnecessary loops". $\endgroup$
    – knb
    Aug 13, 2017 at 16:45

2 Answers 2

2
$\begingroup$

Bayesian models specify priors to inform and constrain the models and get uncertainty estimation in form of a posterior distribution. Non-Bayesian Deep Learning computes a scalar value for weights and biases at each layer. Bayesian Deep Learning calculates a posterior distribution of weights and biases at each layer which better estimates uncertainty but increases computational cost.

Edward is a Python package for Bayesian inference, including Deep Learning.

$\endgroup$
1
  • $\begingroup$ just curious, since Dropout is already proved to be approximated Bayesian neural network, why do we bother variational inference? $\endgroup$ Jun 7, 2018 at 0:49
1
$\begingroup$

I recommend getting to know PyMC library for Bayesian Inference applied to graphical models but also deep learning. See this article for an implementation of Bayesian MLP in PyMC3.

As you would expect, the MLP weights are assigned a prior Gaussian distribution which gets updated to a posterior after observing the training data. Instead of Stochastic Gradient Descent or Adam optimizer, ADVI variational inference algorithm is used to compute the posterior distributions of all latent variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.