# Neural Network Backpropagation problems

I'm working on an implementation of a neural network so I can really grasp how these magic boxes work. However the neural network I've written code for doesn't work and I think it's due to my implementation of backpropagation

[pseudo code]

'last layer only
for each node (i) in last layer
error(i) = outputError(i)
end for

'all hidden/inner layers
for each layer (i) in layers (not including layer 0 or last)
for each neuron (j) in layer i
for each neuron (k) in layer i+1
error(i)(j) +=error(i+1)(k) * weight(i)(j,k)
end for

errors(i)(j) = errors(i)(j) * activation(i)(j) * (1-activation(i)(j))

for each neuron in layer i+1
deltaWeight(i)(j,k) = -error(i)(k)*activation(i)(k)*learningRate/NumExamples
deltaBias(i)(k) = -error(i)(k) * bias(i)(k)*learningRate/NumExample
end for
end for
end for


Then I update all the weights with this:

weight(i)(j,k) = weight(i)(j,k) + deltaWeight(i)(j,k)


Now the problem I have is that the output doesn't seem to get any better. It certainly changes but it doesn't seem to minimise the cost function at all, does anyone know why?

Here is a very nice blog post of how to do it in R from scratch: ParallelR NNs. Follow some links suggested in this post; they are enlightening. Furthermore, additional posts tackle how you can improve on computation time by using parallel computation on CPUs and GPUs.