# More layers in NN give worse result

So I was working on a classification task with the help of a NN. The data-set was normalised, weights random between 0-1, and all the activations were sigmoid function.

Now, when I used a 2 hidden layer model the accuracy was 50%, whereas, when I used 1 hidden layer model the accuracy was 99%. Isn't this contrary to intuitive understanding about NN's. I knew more layers means better fitting even over-fitting, but apparently something different is happening in this case (maybe the values outputted by the second hidden layer is too small for the output layer to discern). So what exactly am I missing?

• You probably want your weights to have some negative values... try -1 to 1 instead – kbrose Jun 7 '18 at 2:47
• @pcko training set – DuttaA Jun 7 '18 at 4:24
• What's your activation functions for the last layer? If possible share your code.. – Aditya Jun 7 '18 at 8:09
• @Aditya sigmoid ...I have used sigmoid for very layer – DuttaA Jun 7 '18 at 9:38

Maybe you are making a mistake, put your code here. But without seeing your code, these are possible points:

• Vanishing problem, I don't think you this problem due to having a very shallow network. You can change your activation function to relu for avoiding that.
• Covariat shift, What it means is that similar to input features which have to be normalized, the inputs of the deeper layers have to be normalized to. The normalization process is in a way that the inputs to each layer should have a special distribution that does not change. You can use batch normalisation for avoiding that problem.
• Bug in coding, you may have fed the activations of each layer to the next layer in a wrong way or you may have updated the weights not simultaneously If you have not used vectorization. There are also numerous different reasons which may lead to bugs in your code.

• The number of neurons may not be enough for each layer, try to increase them. For understanding the meaning of increasing the layers and the number of neurons in each layer, you can take a look at here.

• Only the second one is feasible since the code works perfectly fine for single layer and also that the gradient for the last layer is almost identical for all connections..Or maybe I wasn't inputting enough significant features, although that shouldn't have been a problem since I tried with even 100 nodes for just 10 inputs – DuttaA Jun 8 '18 at 13:17
• @DuttaA how many training data do you have? – Media Jun 8 '18 at 14:30
• I used about 500 though I have 900 – DuttaA Jun 8 '18 at 14:54
• * I wasn't inputting enough significant features* I don't agree with that if you had reached to 99% accuracy. – Media Jun 8 '18 at 15:20
• @Media in my answer DuttaA claims to have constant accuracy and decreasing loss function. What's your opinion on this? Because, to me, it does not make sense at all. – David Masip Jun 12 '18 at 7:47

Run 'Gradient Checking' to locate place where the error occurs because it looks a usual typical mistake. You shouldn't experience any numerical precision at such a shallow network

However, numerical precision might pop-in during the actual gradient checking. Have a look at this to be warned: link