3
$\begingroup$

So I was working on a classification task with the help of a NN. The data-set was normalised, weights random between 0-1, and all the activations were sigmoid function.

Now, when I used a 2 hidden layer model the accuracy was 50%, whereas, when I used 1 hidden layer model the accuracy was 99%. Isn't this contrary to intuitive understanding about NN's. I knew more layers means better fitting even over-fitting, but apparently something different is happening in this case (maybe the values outputted by the second hidden layer is too small for the output layer to discern). So what exactly am I missing?

Improve this question
$\endgroup$
4
  • $\begingroup$ You probably want your weights to have some negative values... try -1 to 1 instead $\endgroup$
    – kbrose
    Commented Jun 7, 2018 at 2:47
  • $\begingroup$ @pcko training set $\endgroup$
    – DuttaA
    Commented Jun 7, 2018 at 4:24
  • $\begingroup$ What's your activation functions for the last layer? If possible share your code.. $\endgroup$
    – Aditya
    Commented Jun 7, 2018 at 8:09
  • $\begingroup$ @Aditya sigmoid ...I have used sigmoid for very layer $\endgroup$
    – DuttaA
    Commented Jun 7, 2018 at 9:38

3 Answers 3

Reset to default
3
$\begingroup$

Maybe you are making a mistake, put your code here. But without seeing your code, these are possible points:

  • Vanishing problem, I don't think you this problem due to having a very shallow network. You can change your activation function to relu for avoiding that.
  • Covariat shift, What it means is that similar to input features which have to be normalized, the inputs of the deeper layers have to be normalized to. The normalization process is in a way that the inputs to each layer should have a special distribution that does not change. You can use batch normalisation for avoiding that problem.

  • Bug in coding, you may have fed the activations of each layer to the next layer in a wrong way or you may have updated the weights not simultaneously If you have not used vectorization. There are also numerous different reasons which may lead to bugs in your code.

  • The number of neurons may not be enough for each layer, try to increase them. For understanding the meaning of increasing the layers and the number of neurons in each layer, you can take a look at here.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Only the second one is feasible since the code works perfectly fine for single layer and also that the gradient for the last layer is almost identical for all connections..Or maybe I wasn't inputting enough significant features, although that shouldn't have been a problem since I tried with even 100 nodes for just 10 inputs $\endgroup$
    – DuttaA
    Commented Jun 8, 2018 at 13:17
  • $\begingroup$ @DuttaA how many training data do you have? $\endgroup$
    – Green Falcon
    Commented Jun 8, 2018 at 14:30
  • $\begingroup$ I used about 500 though I have 900 $\endgroup$
    – DuttaA
    Commented Jun 8, 2018 at 14:54
  • $\begingroup$ * I wasn't inputting enough significant features* I don't agree with that if you had reached to 99% accuracy. $\endgroup$
    – Green Falcon
    Commented Jun 8, 2018 at 15:20
  • $\begingroup$ @Media in my answer DuttaA claims to have constant accuracy and decreasing loss function. What's your opinion on this? Because, to me, it does not make sense at all. $\endgroup$
    – David Masip
    Commented Jun 12, 2018 at 7:47
1
$\begingroup$

Run 'Gradient Checking' to locate place where the error occurs because it looks a usual typical mistake. You shouldn't experience any numerical precision at such a shallow network

However, numerical precision might pop-in during the actual gradient checking. Have a look at this to be warned: link

Also, your weights might be too big - read about Xavier init

Improve this answer
$\endgroup$
1
  • $\begingroup$ No errors ...I always run gradient checking $\endgroup$
    – DuttaA
    Commented Jun 7, 2018 at 4:24
1
$\begingroup$

Training a nn is not always easy. If your traning accuracy is only 50% it means that your network is not really learning. Many problems may arise, but with such a simple problem I would bet that you are having a vanishing gradient problem. If you try activating your hidden layers with relu's, you might solve your vanishing gradient problem. I think it is worth giving a try.

If you are running your nn with Tensorflow, a way to check if your gradients are 0 is using Tensorboard.

Improve this answer
$\endgroup$
6
  • $\begingroup$ You maybe right..Because the very first gradient for the last layer has same weights for all of the connections...Yet the cost decreases quite nicely $\endgroup$
    – DuttaA
    Commented Jun 7, 2018 at 9:39
  • $\begingroup$ Does the cost decrease and the accuracy stays constant? $\endgroup$
    – David Masip
    Commented Jun 7, 2018 at 9:46
  • $\begingroup$ Yep... Pretty much $\endgroup$
    – DuttaA
    Commented Jun 7, 2018 at 11:20
  • $\begingroup$ Looks like a bug on how you measure accuracy. Are you doing batch gradient descent? $\endgroup$
    – David Masip
    Commented Jun 7, 2018 at 14:16
  • $\begingroup$ Yes... How can there be a bug if single layer is working perfectly fine $\endgroup$
    – DuttaA
    Commented Jun 7, 2018 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.