My neural network 'learns' to estimate each class equally

Question

I've implement a neural network. It's a fully connected back propogation network. I'm training it on this data set;

Iris Data Set

I'm using that data just as a test to check if my implimentation is working.

My neural network minimises it's cost function by giving an estimate of each class equally. In this case (Iris) there are three classes and after around 1000 iterations my network output is something like this;

0.3563204029871746
0.34553220619664271
0.36660774391053058

If I run it for 2000 iterations I get this;

0.33748041999368034
0.33555095097164173
0.33929502380142113

The network is almost certainly approaching 1/3 estimate for each class. This is obviously not what is desired, but I can't figure out where my algorithm is going wrong. I can post all my code/pseudocode if it would be useful, but I'm hoping someone has seen this before and can tell me where I'm missing something.

If it matters I'm using a logistic activation function.

Edit: Pseudocode

Ok here's my code;

**Sub DataInput**
Load in the iris text file
split file into lines
redim x(#lines)
redim y(#lines)
for each line
  datain = split line into columns (comma as separator)
  redim x(line #)(3)
  x(0) = datain(0)/7.9
  x(1) = datain(1)/4.4
  x(2).... etc.

  redim y(line #)(2)
  y(0) = 0
  y(1) = 0
  y(2) = 0

  select case datain(5) 'selects which type of flower
    case is flower1
      y(0) = 1
    case is flower2
      y(1) = 1
    case is flower3
      y(2) = 1
  end select
next line
**end sub**

**sub initialise**
'sets up array sizes for all required values
'sets all weights to random
**end sub**

**sub Train**
Do until cost function is below threshold
set DR = 0 for all l, i, j 'DR is the cumulative error value from each training example at each weight (l = layer, i = neuron in next layer, j = neuron in current layer)

for i = 0 to number of training examples
  call sub ForwardPropogate(i)
  call sub BackwardPropogate(i)
end for

for l = 0 to number of layers -1
  for i = 0 to number of neuron in next layer
    for j = 1 to number of neurons in this layer
       CapD(l)(i)(j) = 1/numExamples * DR(l)(i)(j) + lambda*theta(l)(i)(j)
    end for
    CapD(l)(i)(0) = 1/numExamples * DR(l)(i)(0) + lambda*theta(l)(i)(0)
  end for
end for

for l = 0 to number of layers -1
  theta(l) = theta(l) - alpha*CapD(l)
end for
loop

**end sub**

**Sub ForwardPropogate**(byval exNum as int) ' exNum is the training example being propagated

a(0) = x(exNum) ' a(n) is the activations of all neurons in layer n
a(0) = leadingBias(a(0)) ' leading bias adds a 1 to the start of the array for the bias neuron
for i = 1 to number of layers -1
  z(i) = theta(i-1) * a(i-1)
  a(i) = g(z(i)) ' g is the activation function
  a(i) = leadingBias(a(i))
end for
z(number of layers) = theta(number of layers -1) * a(number of layers -1)
a(number of layers) = g(z(number of layers))

**end sub**

**Sub BackwardPropogate**(byval exNum as int)
delta(number of layers) = a(number of layers) - y(exNum)
for i = number of layers - 1 to 1 step -1
  delta(i) = [transpose(theta(i)) * delta(i+1)] .*g'(a(i))    g' is the derivative of the activation function
end for
for i = 0 to number of layers -1
  DR(i) = DR(i) +delta(i+1) * transpose(a(i))
end for

**end sub**

Answer 1

Some things you can try, to help debug your network:

Implemented a gradient check, to make sure you are calculating the gradients correctly.

Experiment with different learning rates, and different initializations of the initial weights.

Make sure to use randomization / symmetry breaking, or some suitable standard method, for initializing the weights.

Try starting with a single layer, and make sure your model can learn (you can solve some simple learning tasks) with a single-layer network. Then try two layers.

Try a ReLU activation function.