neural network not able to learn MNIST dataset in java

so I set off to learn more about AI and I started to create my own AI library. I am starting off with an Artificial neural network since I have created one before in c++ following a tutorial. I believe I implemented it correctly because it can solve the XOR problem. Now when I try to give it the MNIST dataset to learn It doesn't work. All of my output neurons receive at most a 0.1 out of 1 which all map to 0 and hence no number is classified. I have checked my dataset and it's all right. At first, I was not normalizing my inputs but Now I am between 0 and 1(which is how I did it in c++).

I am wondering what is going wrong because I do not think my feedforward or gradient descent is wrong.

Here is some important bits of code:

//Compute the output of an input
public Matrix feedForward(double[] inputs) {
//Initialize the inputs
I.set(new double[][]{inputs});

//Apply to first hidden layer
Z[0] = Matrix.dot(I, W[0]);
H[0] = Z[0].clone();
H[0].applyFunction(activation);

//Apply to rest of hidden layers
for (int i = 1; i < H.length; i++) {
Z[i] = Matrix.dot(H[i - 1], W[i]);
H[i] = Z[i].clone();
H[i].applyFunction(activation);
}

//Apply to output layer
Z[Z.length - 1] = Matrix.dot(H[H.length - 1], W[W.length - 1]);
Z[Z.length - 1].add(B[B.length - 1]);
O = Z[Z.length - 1].clone();
O.applyFunction(activation);
return O;
}

//Apply gradient descent to the network, this will train on one epoch.
public void SGD(double[] target_outputs) {
Matrix.Assert(target_outputs.length == O.getCols(), "Bad Dimensions");
Matrix EO = Matrix.fromArray(new double[][]{target_outputs});

//Get the Errors
Matrix[] Errors = calculateError(EO);

//Apply Error to layers
Matrix dCdW = Matrix.dot(Matrix.transpose(I), Errors[0]);

B[0].subtract(Matrix.multiply(Errors[0], learningRate)); //Adjust Bias

//Rest of network
for (int i = 1; i < Errors.length; i++) {
dCdW = Matrix.dot(Matrix.transpose(H[i - 1]), Errors[i]);

B[i].subtract(Matrix.multiply(Errors[i], learningRate)); //Adjust Bias
}
}

//Calculate the layer error with backpropagation
private Matrix[] calculateError(Matrix target) {
Matrix[] Errors = new Matrix[H.length + 1];

//Calculate output Error
Errors[Errors.length - 1] = O.clone();
Errors[Errors.length - 1].subtract(target);
Errors[Errors.length - 1].hadamard(Matrix.applyFunction(Z[Z.length - 1], activationPrime));

//Calculate hidden errors
for (int i = Errors.length - 2; i >= 0; i--) {
Errors[i] = Matrix.dot(Errors[i + 1], Matrix.transpose(W[i + 1]));
}
return Errors;
}


That is all the important parts of the ANN. In addition, I am using sigmoid as my activation. I have 14 hidden nodes(1 layer) and a learning rate of 0.1 I train it over 30 epochs which I found to be enough in the past. In addition, I am initializing my network with values between 0.001 and 1.

I have been breaking my head over this for days now because I cannot find anything wrong. Any help is appreciated.

Here is some sample output after training: Expected:

{
1x10
0 0 0 0 0 1 0 0 0 0
}


Actual:

{
1x10
0.04583907576231409 0.026962965708026605 0.11717099803714395 0.044286980428971126 0.05459922244196022 0.06290826626128795 0.0890639637876539 0.03994735959422607 0.17140867283797398 0.08934606050531602
}


this is before I apply a step function but it's obvious why this is problematic.

PS: this was copied from another question of mine that i posted on stack overflow.

migrated from ai.stackexchange.comMar 4 at 17:11

This question came from our site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment.

• Which loss function are you using? – DrMcCleod Mar 4 at 7:26
• I believe its a squared loss function so when I take the derivative the 2 cancels out with the 1/2 in the front. Its in my calculate errors function. – Itay Bachar Mar 4 at 21:48