I have a training set, I provide it (consider a data from training set) to the visible layer. Then the normal process is followed, i.e. Positive Phase-> Negative Phase-> Reconstruction of weights, bias units take place. Does it end here or should I have to iterate with the present visible unit value again (the process follows for few fixed iterations) ? Because it is not properly stated anywhere whether with each data of training set, it has to iterated again and again or with each data of training set.


1 Answer 1


When you are unsure about how something (e.g. RBMs) should really be implemented, it is often useful to look at the code of others. G. Hinton himself has published a MATLAB-script (here) which demonstrates the training of an RBM. There, you can see that for each mini-batch, he does the positive phase, then the negative phase, and finally updates the weights - and that's it. So he doesn't iterate between the visible and hidden states.

However, this is not the full truth: for the weight updates we need to know the probability $p(v,h)$. This is very complicated to calculate, as it would contain a sum over all possible states of the RBM. There is a "mathematical trick" called Gibbs sampling: it allows us to iterate back and forth between visible and hidden units to calculate this probability $p(v,h)$.

But: for the result to be correct, we have to iterate forever, which is not really practical. So what Hinton proposed is to iterate for only 1 step instead (this is $CD_1$), so he only goes back-and-forth once.

But, you can also iterate any number of times $k$, which is denoted by $CD_k$. While for Hinton's $CD_1$, you would do

visible --> hidden --> visible

For a $CD_3$, you would iterate from visible to hidden and back, three times:

visible --> hidden --> visible --> hidden --> visible --> hidden --> visible

And just to be clear: you run this iteration for every data point in the training set. Actually, you usually make so-called mini-batches of maybe 10 data points, which you run at the same time. Then you calculate the average weight update from this batch. But then, you do this iteration again, and again, and again, until you have finished your training.

  • $\begingroup$ "calculate the average weight update from this batch"? Calculate the average of each mini-batch or average among the mini-batches? $\endgroup$
    – Born2Code
    Commented Dec 10, 2016 at 2:46
  • $\begingroup$ You do the iteration for one mini-batch, calculate the average update for the e.g. 10 examples from this mini-batch, update the weights, and repeat for the next mini-batch. $\endgroup$
    – hbaderts
    Commented Dec 10, 2016 at 3:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.