1
$\begingroup$

I implemented a fully connected MLP of shape [783 (input), 128 (hidden layer) and 10 (output)] the hidden layer had a sigmoid activation function and the output a sofmax.

I tested with the dataset of keras: Classify images of clothing.

At first I got the ouput was 0.1 at all the exits not matter the input. I then read this and because someone asked about the weights initialization I changed my weights initialization from a normal distribution between [0, 1) to [-1, 1). After that my network started working.

Why did this happen? I believe the prection of 0.1 is some kind of local minima because it just says the same probability for all, at least is what makes sense if you knew nothing about the data. But why? I would love to be refered to a paper that talks about this issue and how to prevent it because I am trying with another dataset now and I got the same problem (but this time I could not make it work. I even try Xavier initialization and still no good result).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Assuming that you normalized the pixel-values as it is done in the tutorial, your inputs were vectors of numbers betweeen 0 and 1. Now, if your weight matrices are also randomly drawn numbers between 0 and 1, the input to the hidden layer will be sums of 783 numbers between 0 and 1, i.e. probably something > 100. Now, check out the sigmoid function and its derivative

enter image description here

As you see, it saturates quite quickly for input values > 5. If you choose the initialization as you did, all hidden neurons should be very close to one, while at the same time the derivative should be very close to zero. This would explain how all outputs of the softmax were equally close to 0.1 and since the gradient was close to zero, the network probably didn't learn anything.

Once you changed the weight initialization to something between -1 and 1, the inputs to the hidden layer should be sums of numbers fairly evenly distributed around 0, thus the sigmoid output was about 0.5 and, most importantly, the gradient was non-zero so your network actually got trained.

As you already noticed, choosing the initialization wisely is crucial for getting decent results. Initializations should also take the number of input neurons into account, otherwise you initialize the network in such a way, that the gradients will be close to zero.

A similar problem can occur if you do not normalize the input data properly, e.g. if you feed the pixel-values between 0 and 255 directly to the model.

I'm not sure about papers on this, but maybe start with the original work on Glorot initialization or checkout the initializers Tensorflow has to offer. On that site under "Functions" they list common initializers and also link to the respective papers.

I hope this was helpful.

Improve this answer
$\endgroup$
3
  • $\begingroup$ The comment was very helpfull and also the paper you cited explained very good the problem as well. Thank you. Marked as solution. $\endgroup$
    – Agustin Barrachina
    Oct 22, 2019 at 13:41
  • $\begingroup$ PS: actually the problem I had on the other dataset was because I forgot to normalize. Solution x2 $\endgroup$
    – Agustin Barrachina
    Oct 22, 2019 at 13:59
  • $\begingroup$ Great then, cheers! $\endgroup$
    – matthiaw91
    Oct 22, 2019 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.