3
$\begingroup$

I am working on solving the handwritten digit recognition problem by implementing a neural network. But the accuracy of the network is coming out to be very low, around 11% for the train dataset. I am not sure what is wrong with my program. I tried changing the learning rate and the number of hidden units, but no luck. Could anyone please take a look and help me out with what I am missing? I am pasting my Julia code below:

# install
Pkg.add("MNIST");
using MNIST

# training data
X,y = traindata(); 
m = size(X, 2);
inputLayerSize = size(X,1); 
hiddenLayerSize = 300;
outputLayerSize = 10;

# representing each output as an array of size of the output layer
eyeY = eye(outputLayerSize);
intY = [convert(Int64,i)+1 for i in y];
Y = zeros(outputLayerSize, m);
for i = 1:m
    Y[:,i] = eyeY[:,intY[i],];
end

# weights with bias
Theta1 = randn(inputLayerSize+1, hiddenLayerSize); 
Theta2 = randn(hiddenLayerSize+1, outputLayerSize); 

function sigmoid(z)
    g = 1.0 ./ (1.0 + exp(-z));
    return g;
end

function sigmoidGradient(z)
  return sigmoid(z).*(1-sigmoid(z));
end

# learning rate
alpha = 0.01;
# number of iterations
epoch = 20;
# cost per epoch
J = zeros(epoch,1);
# backpropagation algorithm
for i = 1:epoch
    for j = 1:m # for each input
        # Feedforward
        # input layer
        # add one bias element
        x1 = [1, X[:,j]];

        # hidden layer
        z2 = Theta1'*x1;
        x2 = sigmoid(z2);
        # add one bias element
        x2 = [1, x2];

        # output layer
        z3 = Theta2'*x2;
        x3 = sigmoid(z3);

        # Backpropagation process
        # delta for output layer
        delta3 = x3 - Y[:,j];
        delta2 = (Theta2[2:end,:]*delta3).*sigmoidGradient(z2) ;

        # update weights
        Theta1 = Theta1 - alpha* x1*delta2';
        Theta2 = Theta2 - alpha* x2*delta3';
    end
end

function predict(Theta1, Theta2, X)
    m = size(X, 2); 
    p = zeros(m, 1);
    h1 = sigmoid(Theta1'*[ones(1,size(X,2)), X]);
    h2 = sigmoid(Theta2'*[ones(1,size(h1,2)), h1]);
    # 1 index is for 0, 2 for 1 ...so forth
    for i=1:m
        p[i,:] = indmax(h2[:,i])-1;
    end
    return p;
end

function accuracy(truth, prediction)
    m = length(truth);
    sum =0;
    for i=1:m
        if truth[i,:] == pred[i,:]
            sum = sum +1;
        end
    end
  return (sum/m)*100;
end

pred = predict(Theta1, Theta2, X);
println("train accuracy: ", accuracy(y, pred));
$\endgroup$
  • $\begingroup$ 11% accuracy is same as random guessing, or simply guessing same value each time. I cannot see any obvious bug. What is size of training set, and have you looked at your learning curve (the J values over time)? It is definitely worth checking your expansion of y into Y is correct - it looks over-complex, although could well be correct, I cannot tell. I might have instead just do something like Y[y[i],i] =1; for simplicity. $\endgroup$ – Neil Slater Aug 14 '15 at 7:46
  • $\begingroup$ Have you verified that back propagation is working as you might expect on a small network and simple problem like XOR ? $\endgroup$ – image_doctor Aug 14 '15 at 8:13
  • 1
    $\begingroup$ I've just ran your code with 10 iterations and got 70.58% of correct results on the training set. Can you repeat your experiment? Also, tracking error after each iteration (essentially, filling up your J array) may help to debug the issue if any. $\endgroup$ – ffriend Aug 14 '15 at 14:55
  • $\begingroup$ @friend You are right. It must be an after effects of late night coding that I mentioned 11% error. After running, 20 iterations the accuracy is 61.133333333333326 and the cost array looks like: 20x1 Array{Float64,2}: 4.14202 4.65382 4.02049 4.57622 4.55148 5.61416 5.51633 4.70868 4.68755 5.10752 4.79347 5.51952 5.05628 5.04076 5.07781 4.9929 5.01385 4.80254 5.20314 4.9887 $\endgroup$ – lex Aug 15 '15 at 7:04
  • 1
    $\begingroup$ Without analyzing the whole code I'd say you don't use specified cost function for optimization, but only for error calculation. If this is the case, gradient descent does a fair job optimizing non-regularized function and allowing coefficients in Theta (and thus error given by costFunction) to grow. I just used simpler error function (J[i] += sum(delta3 .^ 2)) and error seems to constantly decrease (though computation is not finished yet on my machine). $\endgroup$ – ffriend Aug 16 '15 at 23:36
1
$\begingroup$

What loss function are you using? It looks to me like you're using squared error loss (am I right?). This might work, but consider using cross entropy loss instead, which is more well suited for classification problems.

Also, by using the logistic function as activation function in the last layer, you're treating the problem as ten separate binary classification problems. While this also might work, since the problem is a multi-class classification problem, you should probably change the activation function in the last layer to softmax.

One error I spot in your code is that you're not taking into account the derivative of the nonlinearity in the last layer. To fix this, you should change

delta3 = x3 - Y[:,j];

to

delta3 = (x3 - Y[:,j]) .* sigmoidGradient(z3);

similar to the way you calculate delta2.

| improve this answer | |
$\endgroup$
  • $\begingroup$ The OP doesn't calculate loss, but code seems consistent with binary cross entropy loss and thus does not need to adjust for sigmoid derivatives, as they cancel out in the output layer. $\endgroup$ – Neil Slater Jul 28 '17 at 6:59
  • $\begingroup$ @NeilSlater Yes, you're right! $\endgroup$ – HelloGoodbye Jul 28 '17 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.