What is the role hidden vectors play in RBM?

Question

I'm learning about the restricted Boltzmann machine, but I can't decide whether the hidden vectors of RBM are useless or not.

The following are my two understandings for the role hidden vectors play. Have I misunderstood something? If not, then how could the role those vectors play be so different?

1. This is the first understanding:

   I learned from this video RBM that hidden units are just structural support and we don't care about what those hidden vectors really are. We introduce hidden units into RBM just for it to gain more expressive power for probability distribution.

   Let's say I have only one image, and I transform this image to a binary vector and feed this vector into a RBM with random variables (all the weights and biases are chosen randomly). Then, by turning on the machine, the first hidden vector would be constructed. But this hidden vector does not tell us anything, it would be only used to reconstruct a visible vector.

   (my understanding for this reconstructed visible vector is this vector is a vector encoded in the defined RBM in the first place, we are not really construct something new, but we just happen to sample this vector from the defined RBM)

   And we just run this loop of construction and reconstruction for infinitely many times. Finally, what we will get is just the probability distribution encoded in this RBM with random variables.

2. My second understanding goes like this:

   RBM can be used to perform dimensionality reduction, and those hidden vectors are some abstract representations of the raw inputs:

   Given a RBM, each hidden units of the RBM would be a classifier, and what it does is to check the input vector lies in which side of the hyperplane defined by this hidden unit(by the weights and bias). So, if we input an image into this RBM, what the RBM will do is to project this input vector onto those hyperplanes defined by all the hidden units. Thus for an input vector, the corresponding hidden vector is very important, it is some abstract representation. And we can further feed this representation into other models for doing classification.

So, these are my understandings, if you can answer the question by explaining how RBM is used for MNIST that would be extremely helpful for me.

Answer 1

From my point of view, both answers are correct - it depends on what you want to achieve.

One way to use an RBM is as a generative model. If you wanted to do this, you would train the network in order to obtain weights generating a distribution the network which is as close as possible to the real distribution of the input data (not exactly true as CD is known not to converge to the sample distribution, but this just as an aside). You could then, for instance, reconstruct images from partial or sample completely new images. In this context, the main purpose of the hidden units is to increase the power of the network - a Boltzmann machine without hidden units is not able to distinguish target distributions with the same second moments. However, the actual value of the hidden units during and after is less important, as your primary goal is to generate samples.

If you want to use the model to detect features, the hidden units have an additional role. To explain this, suppose you use contrastive divergence as a learning algorithm. In each iteration, you would then sample the hidden units from the visible units and the visible units from the new values of the hidden units. If the result is close to the original value of the visible units, i.e. if your reconstruction error is small, then, apparently, the hidden units and the weights have learned enough about the training data to reconstruct the value of the visible units from the value of the hidden units. As you say, this is in fact similar to a dimensional reduction or an autoencoder. You could then build a second RBM on top the hidden units, i.e. you could start to stack RBMs to build deep networks, where each new layer of hidden units learns more and more abstract features from the training data.