Neural network for MNIST: very low accuracy

I am working on solving the handwritten digit recognition problem by implementing a neural network. But the accuracy of the network is coming out to be very low, around 11% for the train dataset. I am not sure what is wrong with my program. I tried changing the learning rate and the number of hidden units, but no luck. Could anyone please take a look and help me out with what I am missing? I am pasting my Julia code below:

# install
using MNIST

# training data
X,y = traindata();
m = size(X, 2);
inputLayerSize = size(X,1);
hiddenLayerSize = 300;
outputLayerSize = 10;

# representing each output as an array of size of the output layer
eyeY = eye(outputLayerSize);
intY = [convert(Int64,i)+1 for i in y];
Y = zeros(outputLayerSize, m);
for i = 1:m
Y[:,i] = eyeY[:,intY[i],];
end

# weights with bias
Theta1 = randn(inputLayerSize+1, hiddenLayerSize);
Theta2 = randn(hiddenLayerSize+1, outputLayerSize);

function sigmoid(z)
g = 1.0 ./ (1.0 + exp(-z));
return g;
end

return sigmoid(z).*(1-sigmoid(z));
end

# learning rate
alpha = 0.01;
# number of iterations
epoch = 20;
# cost per epoch
J = zeros(epoch,1);
# backpropagation algorithm
for i = 1:epoch
for j = 1:m # for each input
# Feedforward
# input layer
x1 = [1, X[:,j]];

# hidden layer
z2 = Theta1'*x1;
x2 = sigmoid(z2);
x2 = [1, x2];

# output layer
z3 = Theta2'*x2;
x3 = sigmoid(z3);

# Backpropagation process
# delta for output layer
delta3 = x3 - Y[:,j];

# update weights
Theta1 = Theta1 - alpha* x1*delta2';
Theta2 = Theta2 - alpha* x2*delta3';
end
end

function predict(Theta1, Theta2, X)
m = size(X, 2);
p = zeros(m, 1);
h1 = sigmoid(Theta1'*[ones(1,size(X,2)), X]);
h2 = sigmoid(Theta2'*[ones(1,size(h1,2)), h1]);
# 1 index is for 0, 2 for 1 ...so forth
for i=1:m
p[i,:] = indmax(h2[:,i])-1;
end
return p;
end

function accuracy(truth, prediction)
m = length(truth);
sum =0;
for i=1:m
if truth[i,:] == pred[i,:]
sum = sum +1;
end
end
return (sum/m)*100;
end

pred = predict(Theta1, Theta2, X);
println("train accuracy: ", accuracy(y, pred));

• 11% accuracy is same as random guessing, or simply guessing same value each time. I cannot see any obvious bug. What is size of training set, and have you looked at your learning curve (the J values over time)? It is definitely worth checking your expansion of y into Y is correct - it looks over-complex, although could well be correct, I cannot tell. I might have instead just do something like Y[y[i],i] =1; for simplicity. – Neil Slater Aug 14 '15 at 7:46
• Have you verified that back propagation is working as you might expect on a small network and simple problem like XOR ? – image_doctor Aug 14 '15 at 8:13
• I've just ran your code with 10 iterations and got 70.58% of correct results on the training set. Can you repeat your experiment? Also, tracking error after each iteration (essentially, filling up your J array) may help to debug the issue if any. – ffriend Aug 14 '15 at 14:55
• @friend You are right. It must be an after effects of late night coding that I mentioned 11% error. After running, 20 iterations the accuracy is 61.133333333333326 and the cost array looks like: 20x1 Array{Float64,2}: 4.14202 4.65382 4.02049 4.57622 4.55148 5.61416 5.51633 4.70868 4.68755 5.10752 4.79347 5.51952 5.05628 5.04076 5.07781 4.9929 5.01385 4.80254 5.20314 4.9887  – lex Aug 15 '15 at 7:04
• Without analyzing the whole code I'd say you don't use specified cost function for optimization, but only for error calculation. If this is the case, gradient descent does a fair job optimizing non-regularized function and allowing coefficients in Theta (and thus error given by costFunction) to grow. I just used simpler error function (J[i] += sum(delta3 .^ 2)) and error seems to constantly decrease (though computation is not finished yet on my machine). – ffriend Aug 16 '15 at 23:36

What loss function are you using? It looks to me like you're using squared error loss (am I right?). This might work, but consider using cross entropy loss instead, which is more well suited for classification problems.

Also, by using the logistic function as activation function in the last layer, you're treating the problem as ten separate binary classification problems. While this also might work, since the problem is a multi-class classification problem, you should probably change the activation function in the last layer to softmax.

One error I spot in your code is that you're not taking into account the derivative of the nonlinearity in the last layer. To fix this, you should change

delta3 = x3 - Y[:,j];


to

delta3 = (x3 - Y[:,j]) .* sigmoidGradient(z3);


similar to the way you calculate delta2.

• The OP doesn't calculate loss, but code seems consistent with binary cross entropy loss and thus does not need to adjust for sigmoid derivatives, as they cancel out in the output layer. – Neil Slater Jul 28 '17 at 6:59
• @NeilSlater Yes, you're right! – HelloGoodbye Jul 28 '17 at 11:16