So I am writing my own neural network library using back-propagation as my training algorithm. Everything seems fine the error is getting decreased more and more at each iteration however when I am printing the output of the last layer it is definitely not the optimal one! Let me explain it a little further...

My training data is a pretty simple one

Input: {0,0,1},{0,1,1},{1,0,1},{1,1,1},{1,0,0},{1,1,0}
Expected output: {0},{1},{0},{1},{0},{1}

I have tried many variations, even some logical gates (AND, OR, XOR, etc.)! I have tried increasing the number of the neurons on each layer or increasing the layers. I tried both ReLu as the activation function and the common sigmoid one, even some combinations! I have played with the learning rate value but I am still getting the same results. On the above given example of the training set my network's output using ReLu is


and the error is

Error: 2.02248e-06

using the sigmoid function the output is


and the error

Error: 0.0289854

As you can see the expected output of the fourth and sixth element were supposed to be values close to 1 like the second element. I am facing similar problems with different data sets.

So how am I back-propagating the network? I am calculating the output error like this

error = 2.0 * (expected_output - actual_output)

and I am updating the weights like this

w += learning_rate * transposed_layer_output * layer_delta

Am I missing something?


1 Answer 1


I can see a problem with your error function (blue). It is treating under-predictions and over-predictions the same. The error gradient should ideally direct you towards zero error.

error surfaces

For your error function:

$$ E = 2 \cdot (y_{expected} - y_{predicted}) $$

The derivative w.r.t prediction is:

$$ \frac{dE}{d\;y_{predicted}} = -2 $$

Note here that the error is the same whether you under-predict or over predict (i.e. the sign of error doesn't matter in the gradient). So the updates to the weights will be in the same direction (see arrows on blue line).

But if you use a quadratic error function (orange line), over- and under- prediction errors are treated differently (i.e. the gradient depends on the sign of the $expected - predicted$ difference).

$$ E = (y_{expected} - y_{predicted})^2 \\ \frac{dE}{d\;y_{predicted}} = 2 \cdot (y_{expected} - y_{predicted}) \cdot -1 $$

  • $\begingroup$ At the beginning I was like yeah this is right! I need the squared error. But then I figured out that I am actually using mean-squared-error. The error that is using in back-propagation is the derivative of the cost function, right? I am printing the error like (Σ(expected_error - actual_error)²)/n. So the error that I need to back propagate is the derivative of the cost function which is the function that I am using to find the average. This one (expected_error - actual_error)² which derivative is actually the one that I wrote to my question 2.0 * (expected_output - actual_output)right? $\endgroup$ Jul 10, 2019 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.