# How to improve my self-written Neural Network?

I created the following Neural Network in Python. It uses weights and biases which should follow standard procedure.

# Define size of the layers, as well as the learning rate alpha and the max error
inputLayerSize = 2
hiddenLayerSize = 3
outputLayerSize = 1
alpha = 0.5
maxError = 0.001

# Import dependencies
import numpy
from sklearn import preprocessing

# Make random numbers predictable
numpy.random.seed(1)

# Define our activation function
# In this case, we use the Sigmoid function
def sigmoid(x):
output = 1/(1+numpy.exp(-x))
return output
def sigmoid_derivative(x):
return x*(1-x)

# Define the cost function
def calculateError(Y, Y_predicted):
totalError = 0
for i in range(len(Y)):
totalError = totalError + numpy.square(Y[i] - Y_predicted[i])

# Set inputs
# Each row is (x1, x2)
X = numpy.array([
[7, 4.7],
[6.3, 6],
[6.9, 4.9],
[6.4, 5.3],
[5.8, 5.1],
[5.5, 4],
[7.1, 5.9],
[6.3, 5.6],
[6.4, 4.5],
[7.7, 6.7]
])

# Normalize the inputs
#X = preprocessing.scale(X)

# Set goals
# Each row is (y1)
Y = numpy.array([
[0],
[1],
[0],
[1],
[1],
[0],
[0],
[1],
[0],
[1]
])

# Randomly initialize our weights with mean 0
weights_1 = 2*numpy.random.random((inputLayerSize, hiddenLayerSize)) - 1
weights_2 = 2*numpy.random.random((hiddenLayerSize, outputLayerSize)) - 1

# Randomly initialize our bias with mean 0
bias_1 = 2*numpy.random.random((hiddenLayerSize)) - 1
bias_2 = 2*numpy.random.random((outputLayerSize)) - 1

# Loop 10,000 times
for i in xrange(100000):

# Feed forward through layers 0, 1, and 2
layer_0 = X
layer_1 = sigmoid(numpy.dot(layer_0, weights_1)+bias_1)
layer_2 = sigmoid(numpy.dot(layer_1, weights_2)+bias_2)

# Calculate the cost function
# How much did we miss the target value?
layer_2_error = layer_2 - Y

# In what direction is the target value?
# Were we really sure? if so, don't change too much.
layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)

# How much did each layer_1 value contribute to the layer_2 error (according to the weights)?
layer_1_error = layer_2_delta.dot(weights_2.T)

# In what direction is the target layer_1?
# Were we really sure? If so, don't change too much.
layer_1_delta = layer_1_error * sigmoid_derivative(layer_1)

# Update the weights
weights_2 -= alpha * layer_1.T.dot(layer_2_delta)
weights_1 -= alpha * layer_0.T.dot(layer_1_delta)

# Update the bias
bias_2 -= alpha * numpy.sum(layer_2_delta, axis=0)
bias_1 -= alpha * numpy.sum(layer_1_delta, axis=0)

# Print the error to show that we are improving
if (i% 1000) == 0:
print "Error after "+str(i)+" iterations: " + str(calculateError(Y, layer_2))

# Exit if the error is less than maxError
if(calculateError(Y, layer_2)<maxError):
print "Goal reached after "+str(i)+" iterations: " + str(calculateError(Y, layer_2)) + " is smaller than the goal of " + str(maxError)
break

# Show results
print ""
print "Weights between Input Layer -> Hidden Layer"
print weights_1
print ""

print "Bias of Hidden Layer"
print bias_1
print ""

print "Weights between Hidden Layer -> Output Layer"
print weights_2
print ""

print "Bias of Output Layer"
print bias_2
print ""

print "Computed probabilities for SALE (rounded to 3 decimals)"
print numpy.around(layer_2, decimals=3)
print ""

print "Real probabilities for SALE"
print Y
print ""

print "Final Error"
print str(calculateError(Y, layer_2))


Using 32,000 epochs I manage to get on average a final error of 0.001.

However, compared to the MLPClassifier (Scikit-Learn package) using the same parameters:

mlp = MLPClassifier(
hidden_layer_sizes=(3,),
max_iter=32000,
activation='logistic',
tol=0.00001,
verbose='true')


My result is pretty bad. The MLPClassifier gets a final error of 0 when I run it on the same data, after about 10,000 epochs. For both networks I use an input layer size of 2, hidden layer size of 3 and an output layer of 1.

Why does my network need that many more epochs to train? Am I missing an important part?

• I ran your code and got: Final Error [ 0.00389593]. Looks like it is working to me (you have the gradient sign wrong, but then you 'gradient ascent' so that cancels out). If I set alpha = 1.0 then it converges below your error target before iteration 10,000. Want to confirm your test is running on the same code as you present here? – Neil Slater Jul 7 '17 at 20:42
• Hi @NeilSlater Sorry, I had a mistake in my code. I don't want to normalize the data further so I removed the line X = preprocessing.scale(X). If you run the code now, you run into the problem I described. I edited the post and removed the line. – user8272359 Jul 7 '17 at 20:47
• Yes I can replicate your problem now. However, Normalising the data is usually required for neural network. It is not surprising to run into problems if you do not. – Neil Slater Jul 7 '17 at 20:49
• @NeilSlater but shouldn't the bias be able to work with this? You see, the data has already been normalized in the first place. If I normalize the data as before, I can even leave out the weights altogether and get good results. – user8272359 Jul 7 '17 at 20:52
• The bias has nothing to do with it, NNs just work better when fed inputs that are constrained suitable range. Mean 0, SD 1 is best, which is what X = preprocessing.scale(X) does. Give your network a lot more iterations and it will cope with your non-optimally scaled inputs. When I tried it took 209,000 iterations to hit your target error rate, and was a little unstable on the way. Scaling your inputs is important. – Neil Slater Jul 7 '17 at 21:04

### A note on gradient direction

As an aside,

layer_2_error = Y - layer_2


Should be

layer_2_error = layer_2 - Y


weights_2 += -alpha * layer_1.T.dot(layer_2_delta)


This makes no difference to the performance of your network, but I assume you have done this in the rest of the answer.

## Scaling Input to Normalised Values

One clue to performance problems was in your first version which included the code:

X = preprocessing.scale(X)


With this included before training, then the inputs were scaled nicely for working with neural networks and the network converged quickly. Without this, then the network will still operate, but converges much more slowly. Increasing the max iterations to 1,000,000, then I get the result:

Goal reached after 209237 iterations: [ 0.00099997] is smaller than the goal of 0.001

You mention that you don't want to do scale the input, but really in general for NNs you should. It is worth looking at other differences though, because the lack of scaling does not prevent the MLPClassifier from converging.

This is one you spotted and mentioned in the comments. Your bias update is the same for each bias value. To correct this, you want something like:

# Update the bias
bias_2 += -alpha * numpy.sum(layer_2_delta, axis=0)
bias_1 += -alpha * numpy.sum(layer_1_delta, axis=0)


NB - I have assumed you have fixed the gradient direction here.

## Classification Loss Function

MLPClassifier is running a classifier using logloss, which is more efficient loss function that the mean squared error you are using, if your targets are class probabilities. You can use this too, simply by changing:

layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)


To

layer_2_delta = layer_2_error


This is the correct delta value to match the loss function

def calculateError(Y, Y_predicted):
return -numpy.mean( (Y * numpy.log(Y_predicted) +
(1-Y)* numpy.log(1 - Y_predicted)))

• but only when your output layer is sigmoid (the sigmoid derivative cancels out the derivatives of this log function).

• if you use this, you will want to reduce the learning rate for numerical stability. I suggest e.g. alpha = 0.01

• note you can still report your other loss function, for comparison with previous results. Just be aware that you are optimising the log loss

## Difference in Optimisers

You are running batched Stochastic Gradient Descent, which is the most basic optimiser type. The MLPClassifier is using Adam, which is faster and more robust. If you want to compete with it on even terms you need to implement a better optimiser. The simplest improvement is probably to add some momentum.

• Hi, thanks a lot for your great answer! I have ediated my code accordingly. However, it seems like it does not converge anymore - it is stuck at 0.69314716. Any idea why that might be (I think it happens after changing the classification loss function - putting my original classification loss function in it, it converges to 0.001 after 38'411 epochs)? I have defined the seed so we can compare results. See my code here: gist.github.com/TPME/ea9e3a879d2368603dfffd110c560e38 – user8272359 Jul 7 '17 at 22:29
• See my update - your alpha is too high to converge on the new loss function – Neil Slater Jul 7 '17 at 22:37
• Thanks, I changed it. Interestingly, I still get worse results (it does not converge). See: gist.github.com/TPME/4786ad5c967f5e8e319a4dac41ad0383 – user8272359 Jul 7 '17 at 22:41
• I have a copy of your code working nicely (well, -ish, scaling still makes a big difference, it converges in < 1000 iterations with scaling, or 64,331 without). Gist gist.github.com/neilslater/519a153b27df51f124e0eb4f13d89814 – Neil Slater Jul 7 '17 at 22:48
• alpha = 0.05 is better, too, converged in 33345 iterations – Neil Slater Jul 7 '17 at 22:50