8
$\begingroup$

enter image description here

These are 4 different weight matrices that I got after training a restricted Boltzman machine (RBM) with ~4k visible units and only 96 hidden units/weight vectors. As you can see, weights are extremely similar - even black pixels on the face are reproduced. The other 92 vectors are very similar too, though none of weights are exactly the same.

I can overcome this by increasing number of weight vectors to 512 or more. But I encountered this problem several times earlier with different RBM types (binary, Gaussian, even convolutional), different number of hidden units (including pretty large), different hyper-parameters, etc.

My question is: what is the most likely reason for weights to get very similar values? Do they all just get to some local minimum? Or is it a sign of overfitting?

I currently use a kind of Gaussian-Bernoulli RBM, code may be found here.

UPD. My dataset is based on CK+, which contains > 10k images of 327 individuals. However I do pretty heavy preprocessing. First, I clip only pixels inside of outer contour of a face. Second, I transform each face (using piecewise affine wrapping) to the same grid (e.g. eyebrows, nose, lips etc. are in the same (x,y) position on all images). After preprocessing images look like this:

enter image description here enter image description here

When training RBM, I take only non-zero pixels, so outer black region is ignored.

Improve this question
$\endgroup$
9
  • $\begingroup$ Please share some info on the data you are looking at. $\endgroup$
    – bayer
    Commented Aug 17, 2014 at 19:22
  • $\begingroup$ @bayer: please check out my update. $\endgroup$
    – ffriend
    Commented Aug 17, 2014 at 19:48
  • $\begingroup$ This looks like a poor training procedure. Can you add information on the number of CD steps, learning rate/momentum, batch size etc? $\endgroup$
    – bayer
    Commented Aug 18, 2014 at 11:34
  • $\begingroup$ @bayer: at the moment of these experiments I used CD-1, batch size of 10 images, learning rate of 0.01 (0.1 / batch_size) and no momentum at all. I also noticed that weight initialization has some impact: with weights initialized from N(0, 0.01) I have almost never seen described issue, but with weights from N(0, 0.001) I get the issue almost each time. $\endgroup$
    – ffriend
    Commented Aug 18, 2014 at 11:58
  • 1
    $\begingroup$ If your learning rate is too high, the first sample (or the mean of the batch) will be what the RBM overfits to. If the "neurons" (i.e. p(h|v)) then saturate, learning stalls--the gradients of these neurons will be close to zero. This is one way of this happening. $\endgroup$
    – bayer
    Commented Aug 20, 2014 at 15:11

1 Answer 1

Reset to default
1
$\begingroup$

A restricted Boltzmann machine (RBM) learns a lossy compression of the original inputs or in other words, a probability distribution.

Those are 4 different weight matrices are all reduced dimension representations of the original face inputs. If you visualized the weights as a probability distribution, the distributions value would be different but they would have the same amount of loss from the original image reconstruction.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.