0
$\begingroup$

I've implement a neural network. It's a fully connected back propogation network. I'm training it on this data set;

Iris Data Set

I'm using that data just as a test to check if my implimentation is working.

My neural network minimises it's cost function by giving an estimate of each class equally. In this case (Iris) there are three classes and after around 1000 iterations my network output is something like this;

0.3563204029871746
0.34553220619664271
0.36660774391053058

If I run it for 2000 iterations I get this;

0.33748041999368034
0.33555095097164173
0.33929502380142113

The network is almost certainly approaching 1/3 estimate for each class. This is obviously not what is desired, but I can't figure out where my algorithm is going wrong. I can post all my code/pseudocode if it would be useful, but I'm hoping someone has seen this before and can tell me where I'm missing something.

If it matters I'm using a logistic activation function.

Edit: Pseudocode

Ok here's my code;

**Sub DataInput**
Load in the iris text file
split file into lines
redim x(#lines)
redim y(#lines)
for each line
  datain = split line into columns (comma as separator)
  redim x(line #)(3)
  x(0) = datain(0)/7.9
  x(1) = datain(1)/4.4
  x(2).... etc.

  redim y(line #)(2)
  y(0) = 0
  y(1) = 0
  y(2) = 0

  select case datain(5) 'selects which type of flower
    case is flower1
      y(0) = 1
    case is flower2
      y(1) = 1
    case is flower3
      y(2) = 1
  end select
next line
**end sub**

**sub initialise**
'sets up array sizes for all required values
'sets all weights to random
**end sub**

**sub Train**
Do until cost function is below threshold
set DR = 0 for all l, i, j 'DR is the cumulative error value from each training example at each weight (l = layer, i = neuron in next layer, j = neuron in current layer)

for i = 0 to number of training examples
  call sub ForwardPropogate(i)
  call sub BackwardPropogate(i)
end for

for l = 0 to number of layers -1
  for i = 0 to number of neuron in next layer
    for j = 1 to number of neurons in this layer
       CapD(l)(i)(j) = 1/numExamples * DR(l)(i)(j) + lambda*theta(l)(i)(j)
    end for
    CapD(l)(i)(0) = 1/numExamples * DR(l)(i)(0) + lambda*theta(l)(i)(0)
  end for
end for

for l = 0 to number of layers -1
  theta(l) = theta(l) - alpha*CapD(l)
end for
loop

**end sub**

**Sub ForwardPropogate**(byval exNum as int) ' exNum is the training example being propagated

a(0) = x(exNum) ' a(n) is the activations of all neurons in layer n
a(0) = leadingBias(a(0)) ' leading bias adds a 1 to the start of the array for the bias neuron
for i = 1 to number of layers -1
  z(i) = theta(i-1) * a(i-1)
  a(i) = g(z(i)) ' g is the activation function
  a(i) = leadingBias(a(i))
end for
z(number of layers) = theta(number of layers -1) * a(number of layers -1)
a(number of layers) = g(z(number of layers))

**end sub**

**Sub BackwardPropogate**(byval exNum as int)
delta(number of layers) = a(number of layers) - y(exNum)
for i = number of layers - 1 to 1 step -1
  delta(i) = [transpose(theta(i)) * delta(i+1)] .*g'(a(i))    g' is the derivative of the activation function
end for
for i = 0 to number of layers -1
  DR(i) = DR(i) +delta(i+1) * transpose(a(i))
end for

**end sub**
$\endgroup$
  • $\begingroup$ Are the actual predictions accurate? How do you pick the class for classification? $\endgroup$ – HelloWorld Apr 3 '17 at 14:18
  • $\begingroup$ I would check if your backpropagation implementation works, since outputting each class 1/3 of the time is what I'd expect if you randomly initialized the weights and never updated them. $\endgroup$ – liangjy Apr 3 '17 at 14:33
  • $\begingroup$ The output of the network is a vector which should be near 0 everywhere but the index of the correct type of flower. (eg. 1,0,0 or 0,1,0 or 0,0,1). The network does start with random values (and as it happens initially predicts higher values (~0.98) for each class) then over time iterates down to 0.33... $\endgroup$ – FraserOfSmeg Apr 3 '17 at 14:57
  • 1
    $\begingroup$ Would you take a look at iris data-set. It is divided into 3 blocks by species column. Do you think that would help you to figure out why it should go to 1/3? $\endgroup$ – M-- Apr 3 '17 at 21:30
  • $\begingroup$ @Masoud I'm not sure I understand what you're getting at. There are 3 classes of Iris, but for any single line of input my model predicts 0.33 for each class. It's not learning to differentiate between the classes. $\endgroup$ – FraserOfSmeg Apr 4 '17 at 3:33
1
$\begingroup$

Some things you can try, to help debug your network:

Implemented a gradient check, to make sure you are calculating the gradients correctly.

Experiment with different learning rates, and different initializations of the initial weights.

Make sure to use randomization / symmetry breaking, or some suitable standard method, for initializing the weights.

Try starting with a single layer, and make sure your model can learn (you can solve some simple learning tasks) with a single-layer network. Then try two layers.

Try a ReLU activation function.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.