8
$\begingroup$

enter image description here

These are 4 different weight matrices that I got after training a restricted Boltzman machine (RBM) with ~4k visible units and only 96 hidden units/weight vectors. As you can see, weights are extremely similar - even black pixels on the face are reproduced. The other 92 vectors are very similar too, though none of weights are exactly the same.

I can overcome this by increasing number of weight vectors to 512 or more. But I encountered this problem several times earlier with different RBM types (binary, Gaussian, even convolutional), different number of hidden units (including pretty large), different hyper-parameters, etc.

My question is: what is the most likely reason for weights to get very similar values? Do they all just get to some local minimum? Or is it a sign of overfitting?

I currently use a kind of Gaussian-Bernoulli RBM, code may be found here.

UPD. My dataset is based on CK+, which contains > 10k images of 327 individuals. However I do pretty heavy preprocessing. First, I clip only pixels inside of outer contour of a face. Second, I transform each face (using piecewise affine wrapping) to the same grid (e.g. eyebrows, nose, lips etc. are in the same (x,y) position on all images). After preprocessing images look like this:

enter image description here enter image description here

When training RBM, I take only non-zero pixels, so outer black region is ignored.

$\endgroup$
  • $\begingroup$ Please share some info on the data you are looking at. $\endgroup$ – bayer Aug 17 '14 at 19:22
  • $\begingroup$ @bayer: please check out my update. $\endgroup$ – ffriend Aug 17 '14 at 19:48
  • $\begingroup$ This looks like a poor training procedure. Can you add information on the number of CD steps, learning rate/momentum, batch size etc? $\endgroup$ – bayer Aug 18 '14 at 11:34
  • $\begingroup$ @bayer: at the moment of these experiments I used CD-1, batch size of 10 images, learning rate of 0.01 (0.1 / batch_size) and no momentum at all. I also noticed that weight initialization has some impact: with weights initialized from N(0, 0.01) I have almost never seen described issue, but with weights from N(0, 0.001) I get the issue almost each time. $\endgroup$ – ffriend Aug 18 '14 at 11:58
  • 1
    $\begingroup$ If your learning rate is too high, the first sample (or the mean of the batch) will be what the RBM overfits to. If the "neurons" (i.e. p(h|v)) then saturate, learning stalls--the gradients of these neurons will be close to zero. This is one way of this happening. $\endgroup$ – bayer Aug 20 '14 at 15:11
1
$\begingroup$

A restricted Boltzmann machine (RBM) learns a lossy compression of the original inputs or in other words, a probability distribution.

Those are 4 different weight matrices are all reduced dimension representations of the original face inputs. If you visualized the weights as a probability distribution, the distributions value would be different but they would have the same amount of loss from the original image reconstruction.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.