1
$\begingroup$

I'm working on an implementation of a neural network so I can really grasp how these magic boxes work. However the neural network I've written code for doesn't work and I think it's due to my implementation of backpropagation

[pseudo code]

'last layer only
for each node (i) in last layer
error(i) = outputError(i)
end for

'all hidden/inner layers
for each layer (i) in layers (not including layer 0 or last)
 for each neuron (j) in layer i
  for each neuron (k) in layer i+1
   error(i)(j) +=error(i+1)(k) * weight(i)(j,k)
  end for

  errors(i)(j) = errors(i)(j) * activation(i)(j) * (1-activation(i)(j))

  for each neuron in layer i+1
   deltaWeight(i)(j,k) = -error(i)(k)*activation(i)(k)*learningRate/NumExamples
   deltaBias(i)(k) = -error(i)(k) * bias(i)(k)*learningRate/NumExample
  end for
 end for
end for

Then I update all the weights with this:

weight(i)(j,k) = weight(i)(j,k) + deltaWeight(i)(j,k)

Now the problem I have is that the output doesn't seem to get any better. It certainly changes but it doesn't seem to minimise the cost function at all, does anyone know why?

$\endgroup$
1
$\begingroup$

Some reasons the network may not be getting better:

  • Learning rate is too high (jumps back and forth forever)
  • Learning rate is too low (when divided by training set size, goes to zero)
  • You're replacing the error signal with the weight gradient signal. The reuse of error (based on the indentation) seems to be right on the same line as the inner loop. I'm not sure if that's correct. First, backprop your errors, THEN calculate the gradient based on the activation of the hidden neurons and delta sigmoid(error).
| improve this answer | |
$\endgroup$
1
$\begingroup$

Here is a very nice blog post of how to do it in R from scratch: ParallelR NNs. Follow some links suggested in this post; they are enlightening. Furthermore, additional posts tackle how you can improve on computation time by using parallel computation on CPUs and GPUs.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Have a look at Simon Haykin's "Neural Networks and Learning Machines". There is a good treatment of back-propagation there for both sigmoid and radial-basis function networks.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, I'll look into that. A lot of the tutorials I've found online deal with the process in an abstract manner; I'm trying to get into the finer details, will this text help? $\endgroup$ – FraserOfSmeg Mar 11 '16 at 0:28
  • $\begingroup$ The text above has pretty good steps for doing back-propagation. It's spelled out procedure-wise to the point where I got it, which means you should be more than fine. :) $\endgroup$ – JSchultz Mar 14 '16 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.