# MNIST 2-layer NN failing to recognize certain digits

Recently I've been taking a look into some basic multilayer neural networks, and I decided to try tackling the MNIST database of handwritten digits using a 2 layer neural network with 100 neurons in the hidden layer.

My network has a hidden layer using ReLU, and the output layer uses the sigmoid function. The cost is calculated by MSE, and weights are updated using SGD. Below is the code written in MATLAB for training the network.

When training this network, I discovered that it has somehow failed to recognize every 8 and 9 in the data set. It has some problems recognizing other numbers, but not nearly as much as 8 or 9.

How can I fix this issue?

Edit: The results when training a network from a random set of weights are quite inconsistent and oftentimes have low accuracy. Is there something wrong with the backpropagation step or is the network structure not the best for digit recognition?

*dsigmoid and dReLU are the derivatives of their respective functions.

function [trainedWeights1, trainedWeights2] = twoLayerSGD(weights1, weights2, inputs, outputs, alpha, shuffle)
% Trains a 2 layer perceptron using stochastic gradient descent and MSE
%
% function [trainedWeights1, trainedWeights2] = twoLayerSGD(weights2, weights1, inputs, outputs, alpha, numHiddenLayers)
% Inputs:
%   weights1 = weights between input layer and hidden layer as a matrix
%              where row i and column j represents the weight between node
%              j of the input layer and node i of the hidden layer
%   weights2 = weights between hidden layer and output layer as a matrix
%              where row i and column j represents the weight between node j of the
%              input layer and node i of the hidden layer
%   inputs = inputs of the training data, where each set of inputs is
%            stored as a row vector
%   outputs = the correct outputs of the training data, where each set of
%             outputs is stored as a row vector
%   alpha = the learning rate, default is 0.005
%   shuffle = boolean value, determines whether the training cases will be
%             randomly ordered, default is false
%
% Outputs:
%   trainedWeights1 = the trained weight matrix for the weights between the
%                     input layer and hidden layer
%   trainedWeights2 = the trained weight matrix for the weights between the
%                     hidden layer and output layer

if ~exist('alpha','var') || isempty(alpha)
alpha = 0.005;
end

if ~exist('shuffle','var') || isempty(shuffle)
shuffle = false;
end

% Determining the size of the input data set
N = size(inputs,1);
fprintf('Data set size: %.0f\n', N);

% Shuffle input data
if shuffle
fprintf('Shuffling data...\n')
order = randperm(size(inputs, 1));
inputs = inputs(order, :);
outputs = outputs(order, :);
fprintf('Shuffle complete\n')
end

% Train Model
fprintf('Starting training process...\n')
trainedWeights1 = weights1;
trainedWeights2 = weights2;

for dataSet = 1:N
% Get inputs of current data set
input = inputs(dataSet,:); % row vector

% Calculate activations of hidden nodes and the output
Z1 = trainedWeights1*input'; % column vector
hiddenActivation = ReLU(Z1); % column vector
Z2 = trainedWeights2*hiddenActivation; % column vector
output = sigmoid(Z2)'; % row vector

% Calculate the error/cost of the current weights
correctOutput = outputs(dataSet,:); % row vector
cost = sum((output - correctOutput).^2);

% Update Weights
trainedWeights2 = trainedWeights2 - alpha * 2*(output'-correctOutput') .* hiddenActivation' .* dsigmoid(Z2);
S = sum(2*(output'-correctOutput') .* trainedWeights2 .* dsigmoid(Z2),1);
trainedWeights1 = trainedWeights1 - alpha * S' .* input .* dReLU(Z1);

% Display status
if mod(dataSet, 1000) == 0
fprintf('Data set %.0fk/%.0fk complete!\n',dataSet/1000,N/1000);
end
end
end