I've implement a neural network. It's a fully connected back propogation network. I'm training it on this data set;
I'm using that data just as a test to check if my implimentation is working.
My neural network minimises it's cost function by giving an estimate of each class equally. In this case (Iris) there are three classes and after around 1000 iterations my network output is something like this;
0.3563204029871746 0.34553220619664271 0.36660774391053058
If I run it for 2000 iterations I get this;
0.33748041999368034 0.33555095097164173 0.33929502380142113
The network is almost certainly approaching 1/3 estimate for each class. This is obviously not what is desired, but I can't figure out where my algorithm is going wrong. I can post all my code/pseudocode if it would be useful, but I'm hoping someone has seen this before and can tell me where I'm missing something.
If it matters I'm using a logistic activation function.
Ok here's my code;
**Sub DataInput** Load in the iris text file split file into lines redim x(#lines) redim y(#lines) for each line datain = split line into columns (comma as separator) redim x(line #)(3) x(0) = datain(0)/7.9 x(1) = datain(1)/4.4 x(2).... etc. redim y(line #)(2) y(0) = 0 y(1) = 0 y(2) = 0 select case datain(5) 'selects which type of flower case is flower1 y(0) = 1 case is flower2 y(1) = 1 case is flower3 y(2) = 1 end select next line **end sub** **sub initialise** 'sets up array sizes for all required values 'sets all weights to random **end sub** **sub Train** Do until cost function is below threshold set DR = 0 for all l, i, j 'DR is the cumulative error value from each training example at each weight (l = layer, i = neuron in next layer, j = neuron in current layer) for i = 0 to number of training examples call sub ForwardPropogate(i) call sub BackwardPropogate(i) end for for l = 0 to number of layers -1 for i = 0 to number of neuron in next layer for j = 1 to number of neurons in this layer CapD(l)(i)(j) = 1/numExamples * DR(l)(i)(j) + lambda*theta(l)(i)(j) end for CapD(l)(i)(0) = 1/numExamples * DR(l)(i)(0) + lambda*theta(l)(i)(0) end for end for for l = 0 to number of layers -1 theta(l) = theta(l) - alpha*CapD(l) end for loop **end sub** **Sub ForwardPropogate**(byval exNum as int) ' exNum is the training example being propagated a(0) = x(exNum) ' a(n) is the activations of all neurons in layer n a(0) = leadingBias(a(0)) ' leading bias adds a 1 to the start of the array for the bias neuron for i = 1 to number of layers -1 z(i) = theta(i-1) * a(i-1) a(i) = g(z(i)) ' g is the activation function a(i) = leadingBias(a(i)) end for z(number of layers) = theta(number of layers -1) * a(number of layers -1) a(number of layers) = g(z(number of layers)) **end sub** **Sub BackwardPropogate**(byval exNum as int) delta(number of layers) = a(number of layers) - y(exNum) for i = number of layers - 1 to 1 step -1 delta(i) = [transpose(theta(i)) * delta(i+1)] .*g'(a(i)) g' is the derivative of the activation function end for for i = 0 to number of layers -1 DR(i) = DR(i) +delta(i+1) * transpose(a(i)) end for **end sub**