# Troubleshooting Neural Network Implementation

I've been going through the Standford/Coursera Machine Learning course; and it's been going pretty well. I'm really more interested in the understanding of the topic than getting the grade from the course and as such I'm trying to write all the code in a programming language I'm more fluent in (something I can easily dig down into the roots of).

The way I learn best is via working through problems, so I've implemented a neural network and it doesn't work. I seem to get the same probability of each class irrespective of the test example (for example 0.45 of class 0, 0.55 of class 1, irrespective of the input values). Strangely this isn't the case if I remove all the hidden layers.

Here's a brief run through of what I do;

Set all Theta's (weights) to a small random number

for each training example

set activation 0 on layer 0 as 1 (bias)
set layer 1 activations = inputs

forward propagate;
Z(j+1) = Theta(j) x activation(j)  [matrix operations]
activation(j+1) = Sigmoid function (Z(j+1)) [element wise sigmoid]

Set Hx = final layer activations

Set bias of each layer (activation 0,0) = 1

[back propagate]
calculate delta;
delta(last layer) = activation(last layer) - Y  [Y is the expected answer from training set]

delta(j) = transpose(Theta(j)) x delta(j+1) .* (activation(j) .*(Ones - activation(j))

[where ones is a matrix of 1's in every cell; and .* is the element wise multiplication]
[Don't calculate delta(0) since there ins't one for input layer]

DeltaCap(j) = DeltaCap(j) + delta(j+1) x transpose(activation(j))

Next [End for]

Calculate D;

D(j) = 1/#Training * DeltaCap(j) (for j = 0)

D(j) = 1/#Training * DeltaCap(j) + Lambda/#Training * Theta(j) (for j = 0)

[calculate cost function]

J(theta) = -1/#training * Y*Log(Hx) + (1-Y)*log(1-Hx) + lambda/ (2 * #training) * theta^2

Recalculate Theta

Theta = Theta - alpha * D


That's probably not a great deal to go on. If someone can tell me if there's any major flaw in my code that would be fantastic, otherwise some general idea of where I might be going wrong/how to debug something like that that would also be great.

EDIT:

Here's a quick image of the network (including a test case of inputs and responses) (this is after 1 million iterations of gradient descent);

The data set I've used is two exam scores as the x's and the success/failure of getting into a university as the y. Clearly two test scores of 0 would mean failure in getting into university however the network suggests 56% chance of getting it with 0's as inputs.

Edit #2;

I've ran a gradient checking algorithm with the following sort of results;

Numerical calculation: -0.0074962585205895493 Value from propagation: 0.62021047431540277

Numerical calculation: 0.0032635827218463476 Value from propagation: -0.39564819922432665

etc. Clearly there's something wrong here; I'm going to work through it.

• How do you initialise the weights? Have you tried checking the gradients produced by your backprop algorithm (this involves running the feed-forward network many times, with a small variation in each weight and measuring the difference at the output)? – Neil Slater Dec 7 '15 at 8:01
• @NeilSlater I initialise the weights as follows: rnd()×2×e -e [basically picks a small number between -e and e where e is just some near zero value]. I haven't done gradient checking yet, I'm not 100% clear on that process yet. I do plot J(theta) and J(theta)1-J(theta)0 [current J(theta) - last]. – FraserOfSmeg Dec 7 '15 at 8:11
• – Franck Dernoncourt Dec 8 '15 at 14:47
• As far as I see, there is no major flaw in your code. However, many little things can go wrong. I suggest you first try to implement the code using the provided skeleton codes, e.g., ex4.m, etc. In particular, using sigmoidGradient.m can help you check if the your gradient calculation is correct. – yuqian Dec 11 '15 at 3:49