I am working on solving the handwritten digit recognition problem by implementing a neural network. But the accuracy of the network is coming out to be very low, around 11% for the train dataset. I am not sure what is wrong with my program. I tried changing the learning rate and the number of hidden units, but no luck. Could anyone please take a look and help me out with what I am missing? I am pasting my Julia code below:

# install
using MNIST

# training data
X,y = traindata(); 
m = size(X, 2);
inputLayerSize = size(X,1); 
hiddenLayerSize = 300;
outputLayerSize = 10;

# representing each output as an array of size of the output layer
eyeY = eye(outputLayerSize);
intY = [convert(Int64,i)+1 for i in y];
Y = zeros(outputLayerSize, m);
for i = 1:m
    Y[:,i] = eyeY[:,intY[i],];

# weights with bias
Theta1 = randn(inputLayerSize+1, hiddenLayerSize); 
Theta2 = randn(hiddenLayerSize+1, outputLayerSize); 

function sigmoid(z)
    g = 1.0 ./ (1.0 + exp(-z));
    return g;

function sigmoidGradient(z)
  return sigmoid(z).*(1-sigmoid(z));

# learning rate
alpha = 0.01;
# number of iterations
epoch = 20;
# cost per epoch
J = zeros(epoch,1);
# backpropagation algorithm
for i = 1:epoch
    for j = 1:m # for each input
        # Feedforward
        # input layer
        # add one bias element
        x1 = [1, X[:,j]];

        # hidden layer
        z2 = Theta1'*x1;
        x2 = sigmoid(z2);
        # add one bias element
        x2 = [1, x2];

        # output layer
        z3 = Theta2'*x2;
        x3 = sigmoid(z3);

        # Backpropagation process
        # delta for output layer
        delta3 = x3 - Y[:,j];
        delta2 = (Theta2[2:end,:]*delta3).*sigmoidGradient(z2) ;

        # update weights
        Theta1 = Theta1 - alpha* x1*delta2';
        Theta2 = Theta2 - alpha* x2*delta3';

function predict(Theta1, Theta2, X)
    m = size(X, 2); 
    p = zeros(m, 1);
    h1 = sigmoid(Theta1'*[ones(1,size(X,2)), X]);
    h2 = sigmoid(Theta2'*[ones(1,size(h1,2)), h1]);
    # 1 index is for 0, 2 for 1 ...so forth
    for i=1:m
        p[i,:] = indmax(h2[:,i])-1;
    return p;

function accuracy(truth, prediction)
    m = length(truth);
    sum =0;
    for i=1:m
        if truth[i,:] == pred[i,:]
            sum = sum +1;
  return (sum/m)*100;

pred = predict(Theta1, Theta2, X);
println("train accuracy: ", accuracy(y, pred));
  • $\begingroup$ 11% accuracy is same as random guessing, or simply guessing same value each time. I cannot see any obvious bug. What is size of training set, and have you looked at your learning curve (the J values over time)? It is definitely worth checking your expansion of y into Y is correct - it looks over-complex, although could well be correct, I cannot tell. I might have instead just do something like Y[y[i],i] =1; for simplicity. $\endgroup$ Aug 14 '15 at 7:46
  • $\begingroup$ Have you verified that back propagation is working as you might expect on a small network and simple problem like XOR ? $\endgroup$ Aug 14 '15 at 8:13
  • 1
    $\begingroup$ I've just ran your code with 10 iterations and got 70.58% of correct results on the training set. Can you repeat your experiment? Also, tracking error after each iteration (essentially, filling up your J array) may help to debug the issue if any. $\endgroup$
    – ffriend
    Aug 14 '15 at 14:55
  • $\begingroup$ @friend You are right. It must be an after effects of late night coding that I mentioned 11% error. After running, 20 iterations the accuracy is 61.133333333333326 and the cost array looks like: 20x1 Array{Float64,2}: 4.14202 4.65382 4.02049 4.57622 4.55148 5.61416 5.51633 4.70868 4.68755 5.10752 4.79347 5.51952 5.05628 5.04076 5.07781 4.9929 5.01385 4.80254 5.20314 4.9887 $\endgroup$
    – lex
    Aug 15 '15 at 7:04
  • 1
    $\begingroup$ Without analyzing the whole code I'd say you don't use specified cost function for optimization, but only for error calculation. If this is the case, gradient descent does a fair job optimizing non-regularized function and allowing coefficients in Theta (and thus error given by costFunction) to grow. I just used simpler error function (J[i] += sum(delta3 .^ 2)) and error seems to constantly decrease (though computation is not finished yet on my machine). $\endgroup$
    – ffriend
    Aug 16 '15 at 23:36

What loss function are you using? It looks to me like you're using squared error loss (am I right?). This might work, but consider using cross entropy loss instead, which is more well suited for classification problems.

Also, by using the logistic function as activation function in the last layer, you're treating the problem as ten separate binary classification problems. While this also might work, since the problem is a multi-class classification problem, you should probably change the activation function in the last layer to softmax.

One error I spot in your code is that you're not taking into account the derivative of the nonlinearity in the last layer. To fix this, you should change

delta3 = x3 - Y[:,j];


delta3 = (x3 - Y[:,j]) .* sigmoidGradient(z3);

similar to the way you calculate delta2.

  • $\begingroup$ The OP doesn't calculate loss, but code seems consistent with binary cross entropy loss and thus does not need to adjust for sigmoid derivatives, as they cancel out in the output layer. $\endgroup$ Jul 28 '17 at 6:59
  • $\begingroup$ @NeilSlater Yes, you're right! $\endgroup$ Jul 28 '17 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.