Neural network is getting partially trained

Question

So I am writing my own neural network library using back-propagation as my training algorithm. Everything seems fine the error is getting decreased more and more at each iteration however when I am printing the output of the last layer it is definitely not the optimal one! Let me explain it a little further...

My training data is a pretty simple one

Input: {0,0,1},{0,1,1},{1,0,1},{1,1,1},{1,0,0},{1,1,0}
Expected output: {0},{1},{0},{1},{0},{1}

I have tried many variations, even some logical gates (AND, OR, XOR, etc.)! I have tried increasing the number of the neurons on each layer or increasing the layers. I tried both ReLu as the activation function and the common sigmoid one, even some combinations! I have played with the learning rate value but I am still getting the same results. On the above given example of the training set my network's output using ReLu is

0.001585 
0.999990 
0.000713 
0.000109 
0.000000 
0.000000 

and the error is

Error: 2.02248e-06

using the sigmoid function the output is

0.098032 
0.840373 
0.046706 
0.036155 
0.036184 
0.059911 

and the error

Error: 0.0289854

As you can see the expected output of the fourth and sixth element were supposed to be values close to 1 like the second element. I am facing similar problems with different data sets.

So how am I back-propagating the network? I am calculating the output error like this

error = 2.0 * (expected_output - actual_output)

and I am updating the weights like this

w += learning_rate * transposed_layer_output * layer_delta

Am I missing something?

Answer 1

I can see a problem with your error function (blue). It is treating under-predictions and over-predictions the same. The error gradient should ideally direct you towards zero error.

For your error function:

$$ E = 2 \cdot (y_{expected} - y_{predicted}) $$

The derivative w.r.t prediction is:

$$ \frac{dE}{d\;y_{predicted}} = -2 $$

Note here that the error is the same whether you under-predict or over predict (i.e. the sign of error doesn't matter in the gradient). So the updates to the weights will be in the same direction (see arrows on blue line).

But if you use a quadratic error function (orange line), over- and under- prediction errors are treated differently (i.e. the gradient depends on the sign of the $expected - predicted$ difference).

$$ E = (y_{expected} - y_{predicted})^2 \\ \frac{dE}{d\;y_{predicted}} = 2 \cdot (y_{expected} - y_{predicted}) \cdot -1 $$