Hey I am a little new to the whole RBM entropy/energy training thing, having some trouble understanding and trying to figure whether it is worth the effort needed to understand.

Can't RBMs quite easily be trained with a standard gradient descent algorithm? Is the current way just much faster?

Here is some sample Theano code of what I mean:

def rbm(inputs, shape, cost=0, last_cost=1):
    X = T.vector()                     # input
    W = theano.shared(np.zeros(shape)) # weights

    b = theano.shared(np.zeros( shape[1] )) # bias to hidden
    B = theano.shared(np.zeros( shape[0] )) # bias to reconstruction

    H = sigmoid(T.dot(X, W) + b) # hidden activation
    Y = T.dot(H, W.T) + B        # reconstruction

    L = ((X - Y) ** 2).mean()                    # loss function
    G = T.grad(L, W), T.grad(L, b), T.grad(L, B) # gradients

    a = .1                                                  # learning rate
    U = [(W, W - a*G[0]), (b, b - a*G[1]), (B, B - a*G[2])] # shared updates

    train = theano.function([X], L, updates=U) # training function

    while abs(last_cost-cost) > 0.01:
        last_cost, cost = cost, 0
        for input in inputs: cost += train(input)

    return W.get_value(), b.get_value()

And yes this code is very crude.


3 Answers 3


Just some elaborations, RBM is trained to maximize probability of visible units $p(\mathbf{v})$ defined by the model using maximum likelihood, not the reconstruction error, this is true. But the maximum likelihood estimation is performed using Gradient Descent. Another pitfall that analytical expressions for gradients of log-likelihood w.r.t. model parameters are typically intractable (because of exponentially many terms in respective sums), therefore gradients themselves are approximated using MCMC-based algorithm called Contrastive Divergence (or its variants).


RBM is trained using GD/SGD where gradients are approximated using Contrastive Divergence and the function to optimize is log-likelihood of the training data.


What you're trying to do is to minimize the difference between input and its reconstruction. They may be completely different values and RBM may be working fine! It's due to the fact that Restricted Boltzmann Machine models some probabilistic distribution. Learning rule is to find values of weights to model the distribution of the data. You may clamp a data (visible units) and see a reconstruction, but since hidden units are stochastic units, they would have many values. Even when you set your network free and let the units have some values according to stochastic units rule, after some time you can get a sample from RBM, which should be from your data distribution.


The full answer to your question is yes and no. Let me elaborate:

Yes, you can train such a model with SGD:

Yes, you can train a model like you described. This is well known under the name "Autoencoder". The goal of an autoencoder is to find a (usually smaller) representation of the input data, which can then be used to reconstruct the data. This looks as follows:

Structure of an autoencoderImage taken from standford.edu

We try to make $\hat{x}_1$ equal to $x_1$. In your code, you add the additional constraint that $W_1$ (the weights from the input layer to the hidden layer) are equal to $W_2$ (the weights from the hidden layer to the output layer). The reconstruction error is usually either the squared error or the cross-entropy. Such an autoencoder can be trained with SGD, as you proposed.

So, the answer is: yes, what you propose does work and is known as Autoencoder.

No, you can't train an RBM with SGD:

In the previous section, I said that training an RBM with SGD, as described in your question, works. This is not the full truth: this is NOT an RBM anymore!

As described in the answer by Quittend a restricted Boltzmann machine models the probabilistic distribution. The goal of RBM training is not to exactly reconstruct the input, but to learn the probability distribution of the data. The reconstruction is thus not exactly the same as the input, but is a sample from the same probability distribution.

The goal is to assign a high probability to the input data we train on. The probability of an input vector $\mathbf{v}$ is given by $$p(\mathbf{v}) = \frac 1Z \sum_{\mathbf{h}} e^{-E(\mathbf{v}, \mathbf{h})}$$ where $E$ is the energy function, and $Z$ is the partition function (to normalize $p$ such that it is a valid probability). During training, we want to increase $p(\mathbf v)$ for all $\mathbf v$ in our training data, so we have to calculate the derivative $$\frac{\partial \log p(\mathbf v)}{\partial w_{ij}}$$ and change the weights $w_{ij}$ accordingly.

This is what we do with the contrastive divergence (CD-k) algorithm, and this is not possible with gradient descent (SGD).


Training an unsupervised neural network with SGD exists and is known as autoencoder. An RBM is a different concept, which models the probability distribution and does not strictly reconstruct the input. RBMs can not be trained with SGD.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.