Neural Network is not getting better

Question

So i started building my own neural network framework in node.js, just to understand the concept of neural networks better.

Currently i implemented Neurons, Connections between these Neurons and Layers.

Given this network:

N3s output for example, would be Sigmoid((N1 * W1 + N2 * W4) + B) where B is a randomly generated Bias between -1 and 1.

To train the network, a dataset which contains the inputs an their desired output is used. The network will be fed the inputs (Every input getting assigned to one Neuron in the first layer, for example [1, 0] would result in N1 beeing 1 and N2 beeing 0). Then for each Connection, the weight will be increased or decreased by 1% if the resulting output of the network would be closed to the desired one.

For example if 1 is the desired output and the current output would be 0.5, the output with the weight increased would be 0.37 and the output with the weight decreased would be 0.46. The weight would be increased.

Same thing is done for every Neurons Bias.

My Problem:

If i train above network with the dataset:

input: 0, 0
desired Output: 0

input: 0, 1
desired Output: 1

input: 1, 0
desired Output: 1

input: 1, 1
desired Output: 0

for 10000 iterations, it will give me something similar to the following output:

[ 0, 0 ] => [ 0.13500189926566186 ]
[ 0, 1 ] => [ 0.14749124528600177 ]
[ 1, 0 ] => [ 0.14238677719564685 ]
[ 1, 1 ] => [ 0.15412548826116046 ]

I am kinda new to Neural Networks, so sorry if this is a dumb question.

Answer 1

Your training algorithm seems wrong. You should either implement Back Propagation, or tweak your own training algo towards that.

Now your training brings all the weights to an average value, that's why your output seems so close after 10000 iterations. What BP does is it modifies the weights that matter more.

Alternatively, if you don't want to use BP, you can play with evolution instead. It will give you the same result, just slower.

edit: can you post your training algo's code somewhere? Also, you don't really need different objects (like Neurons, Connections, etc.), you only need matrix algebra for an efficient solution.