I've been going through the Standford/Coursera Machine Learning course; and it's been going pretty well. I'm really more interested in the understanding of the topic than getting the grade from the course and as such I'm trying to write all the code in a programming language I'm more fluent in (something I can easily dig down into the roots of).
The way I learn best is via working through problems, so I've implemented a neural network and it doesn't work. I seem to get the same probability of each class irrespective of the test example (for example 0.45 of class 0, 0.55 of class 1, irrespective of the input values). Strangely this isn't the case if I remove all the hidden layers.
Here's a brief run through of what I do;
Set all Theta's (weights) to a small random number for each training example set activation 0 on layer 0 as 1 (bias) set layer 1 activations = inputs forward propagate; Z(j+1) = Theta(j) x activation(j) [matrix operations] activation(j+1) = Sigmoid function (Z(j+1)) [element wise sigmoid] Set Hx = final layer activations Set bias of each layer (activation 0,0) = 1 [back propagate] calculate delta; delta(last layer) = activation(last layer) - Y [Y is the expected answer from training set] delta(j) = transpose(Theta(j)) x delta(j+1) .* (activation(j) .*(Ones - activation(j)) [where ones is a matrix of 1's in every cell; and .* is the element wise multiplication] [Don't calculate delta(0) since there ins't one for input layer] DeltaCap(j) = DeltaCap(j) + delta(j+1) x transpose(activation(j)) Next [End for] Calculate D; D(j) = 1/#Training * DeltaCap(j) (for j = 0) D(j) = 1/#Training * DeltaCap(j) + Lambda/#Training * Theta(j) (for j = 0) [calculate cost function] J(theta) = -1/#training * Y*Log(Hx) + (1-Y)*log(1-Hx) + lambda/ (2 * #training) * theta^2 Recalculate Theta Theta = Theta - alpha * D
That's probably not a great deal to go on. If someone can tell me if there's any major flaw in my code that would be fantastic, otherwise some general idea of where I might be going wrong/how to debug something like that that would also be great.
Here's a quick image of the network (including a test case of inputs and responses) (this is after 1 million iterations of gradient descent);
The data set I've used is two exam scores as the x's and the success/failure of getting into a university as the y. Clearly two test scores of 0 would mean failure in getting into university however the network suggests 56% chance of getting it with 0's as inputs.
I've ran a gradient checking algorithm with the following sort of results;
Numerical calculation: -0.0074962585205895493 Value from propagation: 0.62021047431540277
Numerical calculation: 0.0032635827218463476 Value from propagation: -0.39564819922432665
etc. Clearly there's something wrong here; I'm going to work through it.