2
$\begingroup$

I'm trying to debug why my neural network isn't working. One of the things I've observed is that the weights between the input layer and the first hidden layer hardly change at all, whereas weights later in the network (eg. the weights between the last hidden layer and the output) change significantly. Is this to be expected or a symptom of an error in my code?

I'm applying backpropagation and gradient descent to alter the weights.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

This is expected and well established - Vanished Gradient

https://en.wikipedia.org/wiki/Vanishing_gradient_problem

This has the effect of multiplying n of these small numbers to compute gradients of the "front" layers in an n-layer network, meaning that the gradient (error signal) decreases exponentially with n and the front layers train very slowly.

While I'm not an expert in neural network (experts, please add an answer), I'm sure professional implementation for neural network is tricky and highly optimized. If you simply implement a vanilla text-book-styled neural network, you won't be able to use it to train a large network. Keep it small and you'll be fine.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ I would add that there are 2 things the OP could do: 1) Check their implementation is correctly calculating gradients by implementing gradient testing. 2) Implement and try ReLU activation function, which suffers less from vanishing gradient problems (provided there are enough positive activations). $\endgroup$
    – Neil Slater
    Commented Apr 4, 2017 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.