I've been going through the Standford/Coursera Machine Learning course; and it's been going pretty well. I'm really more interested in the understanding of the topic than getting the grade from the course and as such I'm trying to write all the code in a programming language I'm more fluent in (something I can easily dig down into the roots of).

The way I learn best is via working through problems, so I've implemented a neural network and it doesn't work. I seem to get the same probability of each class irrespective of the test example (for example 0.45 of class 0, 0.55 of class 1, irrespective of the input values). Strangely this isn't the case if I remove all the hidden layers.

Here's a brief run through of what I do;

Set all Theta's (weights) to a small random number

for each training example

set activation 0 on layer 0 as 1 (bias)
set layer 1 activations = inputs

forward propagate;
 Z(j+1) = Theta(j) x activation(j)  [matrix operations]
 activation(j+1) = Sigmoid function (Z(j+1)) [element wise sigmoid]

Set Hx = final layer activations

Set bias of each layer (activation 0,0) = 1

[back propagate]
calculate delta;
delta(last layer) = activation(last layer) - Y  [Y is the expected answer from training set]

delta(j) = transpose(Theta(j)) x delta(j+1) .* (activation(j) .*(Ones - activation(j))

[where ones is a matrix of 1's in every cell; and .* is the element wise multiplication]
[Don't calculate delta(0) since there ins't one for input layer]

DeltaCap(j) = DeltaCap(j) + delta(j+1) x transpose(activation(j))

Next [End for]

Calculate D;

D(j) = 1/#Training * DeltaCap(j) (for j = 0)

D(j) = 1/#Training * DeltaCap(j) + Lambda/#Training * Theta(j) (for j = 0)

[calculate cost function]

J(theta) = -1/#training * Y*Log(Hx) + (1-Y)*log(1-Hx) + lambda/ (2 * #training) * theta^2

Recalculate Theta

Theta = Theta - alpha * D

That's probably not a great deal to go on. If someone can tell me if there's any major flaw in my code that would be fantastic, otherwise some general idea of where I might be going wrong/how to debug something like that that would also be great.


Here's a quick image of the network (including a test case of inputs and responses) (this is after 1 million iterations of gradient descent);

enter image description here

The data set I've used is two exam scores as the x's and the success/failure of getting into a university as the y. Clearly two test scores of 0 would mean failure in getting into university however the network suggests 56% chance of getting it with 0's as inputs.

Edit #2;

I've ran a gradient checking algorithm with the following sort of results;

Numerical calculation: -0.0074962585205895493 Value from propagation: 0.62021047431540277

Numerical calculation: 0.0032635827218463476 Value from propagation: -0.39564819922432665

etc. Clearly there's something wrong here; I'm going to work through it.

  • 1
    $\begingroup$ How do you initialise the weights? Have you tried checking the gradients produced by your backprop algorithm (this involves running the feed-forward network many times, with a small variation in each weight and measuring the difference at the output)? $\endgroup$ – Neil Slater Dec 7 '15 at 8:01
  • $\begingroup$ @NeilSlater I initialise the weights as follows: rnd()×2×e -e [basically picks a small number between -e and e where e is just some near zero value]. I haven't done gradient checking yet, I'm not 100% clear on that process yet. I do plot J(theta) and J(theta)1-J(theta)0 [current J(theta) - last]. $\endgroup$ – FraserOfSmeg Dec 7 '15 at 8:11
  • $\begingroup$ As far as I see, there is no major flaw in your code. However, many little things can go wrong. I suggest you first try to implement the code using the provided skeleton codes, e.g., ex4.m, etc. In particular, using sigmoidGradient.m can help you check if the your gradient calculation is correct. $\endgroup$ – yuqian Dec 11 '15 at 3:49

Are your inputs scaled? Not doing so can cause the weights to immediately blow up.

It's common practice to preprocess your input to be between -1 and 1 or at least in that range. Otherwise you are running the risk of your gradients exploding or vanishing (covered here).

| improve this answer | |
  • $\begingroup$ Thanks for the input (pun intended) but since both inputs are scores out of 100 both features are of the same scale so feature scaling isn't required (at least that's my understanding?). Nevertheless I did think of this as well and I have tried it with each value scaled to between 0 and 1 (score / 100) but the same problems arose. $\endgroup$ – FraserOfSmeg Dec 8 '15 at 8:46
  • $\begingroup$ Have you looked at the activations of each layer for a few iterations? $\endgroup$ – jamesmf Dec 8 '15 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.