so I set off to learn more about AI and I started to create my own AI library. I am starting off with an Artificial neural network since I have created one before in c++ following a tutorial. I believe I implemented it correctly because it can solve the XOR problem. Now when I try to give it the MNIST dataset to learn It doesn't work. All of my output neurons receive at most a 0.1 out of 1 which all map to 0 and hence no number is classified. I have checked my dataset and it's all right. At first, I was not normalizing my inputs but Now I am between 0 and 1(which is how I did it in c++).

I am wondering what is going wrong because I do not think my feedforward or gradient descent is wrong.

Here is some important bits of code:

//Compute the output of an input
public Matrix feedForward(double[] inputs) {
    //Initialize the inputs
    I.set(new double[][]{inputs});

    //Apply to first hidden layer
    Z[0] = Matrix.dot(I, W[0]);
    H[0] = Z[0].clone();

    //Apply to rest of hidden layers
    for (int i = 1; i < H.length; i++) {
        Z[i] = Matrix.dot(H[i - 1], W[i]);
        H[i] = Z[i].clone();

    //Apply to output layer
    Z[Z.length - 1] = Matrix.dot(H[H.length - 1], W[W.length - 1]);
    Z[Z.length - 1].add(B[B.length - 1]);
    O = Z[Z.length - 1].clone();
    return O;

//Apply gradient descent to the network, this will train on one epoch.
public void SGD(double[] target_outputs) {
    Matrix.Assert(target_outputs.length == O.getCols(), "Bad Dimensions");
    Matrix EO = Matrix.fromArray(new double[][]{target_outputs});

    //Get the Errors
    Matrix[] Errors = calculateError(EO);

    //Apply Error to layers
    //Start with first weight and bias
    Matrix dCdW = Matrix.dot(Matrix.transpose(I), Errors[0]);

    W[0].subtract(dCdW); //Adjust Weight
    B[0].subtract(Matrix.multiply(Errors[0], learningRate)); //Adjust Bias

    //Rest of network
    for (int i = 1; i < Errors.length; i++) {
        dCdW = Matrix.dot(Matrix.transpose(H[i - 1]), Errors[i]);

        W[i].subtract(dCdW); //Adjust Weight
        B[i].subtract(Matrix.multiply(Errors[i], learningRate)); //Adjust Bias

//Calculate the layer error with backpropagation
private Matrix[] calculateError(Matrix target) {
    Matrix[] Errors = new Matrix[H.length + 1];

    //Calculate output Error
    Errors[Errors.length - 1] = O.clone();
    Errors[Errors.length - 1].subtract(target);
    Errors[Errors.length - 1].hadamard(Matrix.applyFunction(Z[Z.length - 1], activationPrime));

    //Calculate hidden errors
    for (int i = Errors.length - 2; i >= 0; i--) {
        Errors[i] = Matrix.dot(Errors[i + 1], Matrix.transpose(W[i + 1]));
        Errors[i].hadamard(Matrix.applyFunction(Z[i], activationPrime));
    return Errors;

That is all the important parts of the ANN. In addition, I am using sigmoid as my activation. I have 14 hidden nodes(1 layer) and a learning rate of 0.1 I train it over 30 epochs which I found to be enough in the past. In addition, I am initializing my network with values between 0.001 and 1.

I have been breaking my head over this for days now because I cannot find anything wrong. Any help is appreciated.

Here is some sample output after training: Expected:

0 0 0 0 0 1 0 0 0 0 


0.04583907576231409 0.026962965708026605 0.11717099803714395 0.044286980428971126 0.05459922244196022 0.06290826626128795 0.0890639637876539 0.03994735959422607 0.17140867283797398 0.08934606050531602 

this is before I apply a step function but it's obvious why this is problematic.

PS: this was copied from another question of mine that i posted on stack overflow.


migrated from ai.stackexchange.com Mar 4 at 17:11

This question came from our site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment.

  • $\begingroup$ Which loss function are you using? $\endgroup$ – DrMcCleod Mar 4 at 7:26
  • $\begingroup$ I believe its a squared loss function so when I take the derivative the 2 cancels out with the 1/2 in the front. Its in my calculate errors function. $\endgroup$ – Itay Bachar Mar 4 at 21:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.