I'm working on an implementation of a neural network so I can really grasp how these magic boxes work. However the neural network I've written code for doesn't work and I think it's due to my implementation of backpropagation
'last layer only for each node (i) in last layer error(i) = outputError(i) end for 'all hidden/inner layers for each layer (i) in layers (not including layer 0 or last) for each neuron (j) in layer i for each neuron (k) in layer i+1 error(i)(j) +=error(i+1)(k) * weight(i)(j,k) end for errors(i)(j) = errors(i)(j) * activation(i)(j) * (1-activation(i)(j)) for each neuron in layer i+1 deltaWeight(i)(j,k) = -error(i)(k)*activation(i)(k)*learningRate/NumExamples deltaBias(i)(k) = -error(i)(k) * bias(i)(k)*learningRate/NumExample end for end for end for
Then I update all the weights with this:
weight(i)(j,k) = weight(i)(j,k) + deltaWeight(i)(j,k)
Now the problem I have is that the output doesn't seem to get any better. It certainly changes but it doesn't seem to minimise the cost function at all, does anyone know why?