How to improve my self-written Neural Network?

Question

I created the following Neural Network in Python. It uses weights and biases which should follow standard procedure.

# Define size of the layers, as well as the learning rate alpha and the max error
inputLayerSize = 2
hiddenLayerSize = 3
outputLayerSize = 1
alpha = 0.5
maxError = 0.001

# Import dependencies
import numpy
from sklearn import preprocessing

# Make random numbers predictable
numpy.random.seed(1)

# Define our activation function
# In this case, we use the Sigmoid function
def sigmoid(x):
    output = 1/(1+numpy.exp(-x))
    return output
def sigmoid_derivative(x):
    return x*(1-x)

# Define the cost function
def calculateError(Y, Y_predicted):
    totalError = 0
    for i in range(len(Y)):
        totalError = totalError + numpy.square(Y[i] - Y_predicted[i])
    return totalError

# Set inputs
# Each row is (x1, x2)
X = numpy.array([
            [7, 4.7],
            [6.3, 6],
            [6.9, 4.9],
            [6.4, 5.3],
            [5.8, 5.1],
            [5.5, 4],
            [7.1, 5.9],
            [6.3, 5.6],
            [6.4, 4.5],
            [7.7, 6.7]
            ])

# Normalize the inputs
#X = preprocessing.scale(X)

# Set goals
# Each row is (y1)
Y = numpy.array([
            [0],
            [1],
            [0],
            [1],
            [1],
            [0],
            [0],
            [1],
            [0],
            [1]
            ])

# Randomly initialize our weights with mean 0
weights_1 = 2*numpy.random.random((inputLayerSize, hiddenLayerSize)) - 1
weights_2 = 2*numpy.random.random((hiddenLayerSize, outputLayerSize)) - 1

# Randomly initialize our bias with mean 0
bias_1 = 2*numpy.random.random((hiddenLayerSize)) - 1
bias_2 = 2*numpy.random.random((outputLayerSize)) - 1

# Loop 10,000 times
for i in xrange(100000):

    # Feed forward through layers 0, 1, and 2
    layer_0 = X
    layer_1 = sigmoid(numpy.dot(layer_0, weights_1)+bias_1)
    layer_2 = sigmoid(numpy.dot(layer_1, weights_2)+bias_2)

    # Calculate the cost function
    # How much did we miss the target value?
    layer_2_error = layer_2 - Y

    # In what direction is the target value?
    # Were we really sure? if so, don't change too much.
    layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)

    # How much did each layer_1 value contribute to the layer_2 error (according to the weights)?
    layer_1_error = layer_2_delta.dot(weights_2.T)

    # In what direction is the target layer_1?
    # Were we really sure? If so, don't change too much.
    layer_1_delta = layer_1_error * sigmoid_derivative(layer_1)

    # Update the weights
    weights_2 -= alpha * layer_1.T.dot(layer_2_delta)
    weights_1 -= alpha * layer_0.T.dot(layer_1_delta)

    # Update the bias    
    bias_2 -= alpha * numpy.sum(layer_2_delta, axis=0)
    bias_1 -= alpha * numpy.sum(layer_1_delta, axis=0)

    # Print the error to show that we are improving
    if (i% 1000) == 0:
        print "Error after "+str(i)+" iterations: " + str(calculateError(Y, layer_2))

    # Exit if the error is less than maxError
    if(calculateError(Y, layer_2)<maxError):
        print "Goal reached after "+str(i)+" iterations: " + str(calculateError(Y, layer_2)) + " is smaller than the goal of " + str(maxError)
        break

# Show results
print ""
print "Weights between Input Layer -> Hidden Layer"
print weights_1
print ""

print "Bias of Hidden Layer"
print bias_1
print ""

print "Weights between Hidden Layer -> Output Layer"
print weights_2
print ""

print "Bias of Output Layer"
print bias_2
print ""

print "Computed probabilities for SALE (rounded to 3 decimals)"
print numpy.around(layer_2, decimals=3)
print ""

print "Real probabilities for SALE"
print Y
print ""

print "Final Error"
print str(calculateError(Y, layer_2))

Using 32,000 epochs I manage to get on average a final error of 0.001.

However, compared to the MLPClassifier (Scikit-Learn package) using the same parameters:

mlp = MLPClassifier(
     hidden_layer_sizes=(3,),
     max_iter=32000,
     activation='logistic',
     tol=0.00001,
     verbose='true')

My result is pretty bad. The MLPClassifier gets a final error of 0 when I run it on the same data, after about 10,000 epochs. For both networks I use an input layer size of 2, hidden layer size of 3 and an output layer of 1.

Why does my network need that many more epochs to train? Am I missing an important part?

Answer 1

A note on gradient direction

As an aside,

layer_2_error = Y - layer_2

Should be

layer_2_error = layer_2 - Y

And all your update functions should be gradient descent e.g.

weights_2 += -alpha * layer_1.T.dot(layer_2_delta)

This makes no difference to the performance of your network, but I assume you have done this in the rest of the answer.

Scaling Input to Normalised Values

One clue to performance problems was in your first version which included the code:

X = preprocessing.scale(X)

With this included before training, then the inputs were scaled nicely for working with neural networks and the network converged quickly. Without this, then the network will still operate, but converges much more slowly. Increasing the max iterations to 1,000,000, then I get the result:

Goal reached after 209237 iterations: [ 0.00099997] is smaller than the goal of 0.001

You mention that you don't want to do scale the input, but really in general for NNs you should. It is worth looking at other differences though, because the lack of scaling does not prevent the MLPClassifier from converging.

Bias Gradients

This is one you spotted and mentioned in the comments. Your bias update is the same for each bias value. To correct this, you want something like:

# Update the bias    
bias_2 += -alpha * numpy.sum(layer_2_delta, axis=0)
bias_1 += -alpha * numpy.sum(layer_1_delta, axis=0)

NB - I have assumed you have fixed the gradient direction here.

Classification Loss Function

MLPClassifier is running a classifier using logloss, which is more efficient loss function that the mean squared error you are using, if your targets are class probabilities. You can use this too, simply by changing:

layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)

To

layer_2_delta = layer_2_error

This is the correct delta value to match the loss function

def calculateError(Y, Y_predicted):
    return -numpy.mean( (Y * numpy.log(Y_predicted) + 
       (1-Y)* numpy.log(1 - Y_predicted)))

• but only when your output layer is sigmoid (the sigmoid derivative cancels out the derivatives of this log function).

• if you use this, you will want to reduce the learning rate for numerical stability. I suggest e.g. alpha = 0.01

• note you can still report your other loss function, for comparison with previous results. Just be aware that you are optimising the log loss

Difference in Optimisers

You are running batched Stochastic Gradient Descent, which is the most basic optimiser type. The MLPClassifier is using Adam, which is faster and more robust. If you want to compete with it on even terms you need to implement a better optimiser. The simplest improvement is probably to add some momentum.