Skip to main content
Data Science

Return to Answer

added 147 characters in body
Source Link
Neil Slater
Neil Slater
  • 29.2k
  • 5
  • 81
  • 100

A note on gradient direction

As an aside,

layer_2_error = Y - layer_2

Should be

layer_2_error = layer_2 - Y

And all your update functions should be gradient descent e.g.

weights_2 += -alpha * layer_1.T.dot(layer_2_delta)

This makes no difference to the performance of your network, but I assume you have done this in the rest of the answer.

Scaling Input to Normalised Values

One clue to performance problems was in your first version which included the code:

X = preprocessing.scale(X)

With this included before training, then the inputs were scaled nicely for working with neural networks and the network converged quickly. Without this, then the network will still operate, but converges much more slowly. Increasing the max iterations to 1,000,000, then I get the result:

Goal reached after 209237 iterations: [ 0.00099997] is smaller than the goal of 0.001

You mention that you don't want to do scale the input, but really in general for NNs you should. It is worth looking at other differences though, because the lack of scaling does not prevent the MLPClassifier from converging.

Bias Gradients

This is one you spotted and mentioned in the comments. Your bias update is the same for each bias value. To correct this, you want something like:

# Update the bias    
bias_2 += -alpha * numpy.sum(layer_2_delta, axis=0)
bias_1 += -alpha * numpy.sum(layer_1_delta, axis=0)

NB - I have assumed you have fixed the gradient direction here.

Classification Loss Function

MLPClassifier is running a classifier using logloss, which is more efficient loss function that the mean squared error you are using, if your targets are class probabilities. You can use this too, simply by changing:

layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)

To

layer_2_delta = layer_2_error

This is the correct delta value to match the loss function

def calculateError(Y, Y_predicted):
    return -numpy.mean( (Y * numpy.log(Y_predicted) + 
       (1-Y)* numpy.log(1 - Y_predicted)))

  • but only when your output layer is sigmoid (the sigmoid derivative cancels out the derivatives of this log function).

  • if you use this, you will want to reduce the learning rate for numerical stability. I suggest e.g. alpha = 0.01

  • note you can still report your other loss function, for comparison with previous results. Just be aware that you are optimising the log loss

Difference in Optimisers

You are running batched Stochastic Gradient Descent, which is the most basic optimiser type. The MLPClassifier is using Adam, which is faster and more robust. If you want to compete with it on even terms you need to implement a better optimiser. The simplest improvement is probably to add some momentum.

A note on gradient direction

As an aside,

layer_2_error = Y - layer_2

Should be

layer_2_error = layer_2 - Y

And all your update functions should be gradient descent e.g.

weights_2 += -alpha * layer_1.T.dot(layer_2_delta)

This makes no difference to the performance of your network, but I assume you have done this in the rest of the answer.

Scaling Input to Normalised Values

One clue to performance problems was in your first version which included the code:

X = preprocessing.scale(X)

With this included before training, then the inputs were scaled nicely for working with neural networks and the network converged quickly. Without this, then the network will still operate, but converges much more slowly. Increasing the max iterations to 1,000,000, then I get the result:

Goal reached after 209237 iterations: [ 0.00099997] is smaller than the goal of 0.001

You mention that you don't want to do scale the input, but really in general for NNs you should. It is worth looking at other differences though, because the lack of scaling does not prevent the MLPClassifier from converging.

Bias Gradients

This is one you spotted and mentioned in the comments. Your bias update is the same for each bias value. To correct this, you want something like:

# Update the bias    
bias_2 += -alpha * numpy.sum(layer_2_delta, axis=0)
bias_1 += -alpha * numpy.sum(layer_1_delta, axis=0)

NB - I have assumed you have fixed the gradient direction here.

Classification Loss Function

MLPClassifier is running a classifier using logloss, which is more efficient loss function that the mean squared error you are using, if your targets are class probabilities. You can use this too, simply by changing:

layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)

To

layer_2_delta = layer_2_error

This is the correct delta value to match the loss function

def calculateError(Y, Y_predicted):
    return -numpy.mean( (Y * numpy.log(Y_predicted) + 
       (1-Y)* numpy.log(1 - Y_predicted)))

  • but only when your output layer is sigmoid (the sigmoid derivative cancels out the derivatives of this log function).

  • if you use this, you will want to reduce the learning rate for numerical stability. I suggest e.g. alpha = 0.01

Difference in Optimisers

You are running batched Stochastic Gradient Descent, which is the most basic optimiser type. The MLPClassifier is using Adam, which is faster and more robust. If you want to compete with it on even terms you need to implement a better optimiser. The simplest improvement is probably to add some momentum.

A note on gradient direction

As an aside,

layer_2_error = Y - layer_2

Should be

layer_2_error = layer_2 - Y

And all your update functions should be gradient descent e.g.

weights_2 += -alpha * layer_1.T.dot(layer_2_delta)

This makes no difference to the performance of your network, but I assume you have done this in the rest of the answer.

Scaling Input to Normalised Values

One clue to performance problems was in your first version which included the code:

X = preprocessing.scale(X)

With this included before training, then the inputs were scaled nicely for working with neural networks and the network converged quickly. Without this, then the network will still operate, but converges much more slowly. Increasing the max iterations to 1,000,000, then I get the result:

Goal reached after 209237 iterations: [ 0.00099997] is smaller than the goal of 0.001

You mention that you don't want to do scale the input, but really in general for NNs you should. It is worth looking at other differences though, because the lack of scaling does not prevent the MLPClassifier from converging.

Bias Gradients

This is one you spotted and mentioned in the comments. Your bias update is the same for each bias value. To correct this, you want something like:

# Update the bias    
bias_2 += -alpha * numpy.sum(layer_2_delta, axis=0)
bias_1 += -alpha * numpy.sum(layer_1_delta, axis=0)

NB - I have assumed you have fixed the gradient direction here.

Classification Loss Function

MLPClassifier is running a classifier using logloss, which is more efficient loss function that the mean squared error you are using, if your targets are class probabilities. You can use this too, simply by changing:

layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)

To

layer_2_delta = layer_2_error

This is the correct delta value to match the loss function

def calculateError(Y, Y_predicted):
    return -numpy.mean( (Y * numpy.log(Y_predicted) + 
       (1-Y)* numpy.log(1 - Y_predicted)))

  • but only when your output layer is sigmoid (the sigmoid derivative cancels out the derivatives of this log function).

  • if you use this, you will want to reduce the learning rate for numerical stability. I suggest e.g. alpha = 0.01

  • note you can still report your other loss function, for comparison with previous results. Just be aware that you are optimising the log loss

Difference in Optimisers

You are running batched Stochastic Gradient Descent, which is the most basic optimiser type. The MLPClassifier is using Adam, which is faster and more robust. If you want to compete with it on even terms you need to implement a better optimiser. The simplest improvement is probably to add some momentum.

added 14 characters in body
Source Link
Neil Slater
Neil Slater
  • 29.2k
  • 5
  • 81
  • 100

A note on gradient direction

As an aside,

layer_2_error = Y - layer_2

Should be

layer_2_error = layer_2 - Y

And all your update functions should be gradient descent e.g.

weights_2 += -alpha * layer_1.T.dot(layer_2_delta)

This makes no difference to the performance of your network, but I assume you have done this in the rest of the answer.

Scaling Input to Normalised Values

One clue to performance problems was in your first version which included the code:

X = preprocessing.scale(X)

With this included before training, then the inputs were scaled nicely for working with neural networks and the network converged quickly. Without this, then the network will still operate, but converges much more slowly. Increasing the max iterations to 1,000,000, then I get the result:

Goal reached after 209237 iterations: [ 0.00099997] is smaller than the goal of 0.001

You mention that you don't want to do scale the input, but really in general for NNs you should. It is worth looking at other differences though, because the lack of scaling does not prevent the MLPClassifier from converging.

Bias Gradients

This is one you spotted and mentioned in the comments. Your bias update is the same for each bias value. To correct this, you want something like:

# Update the bias    
bias_2 += -alpha * numpy.meansum(layer_2_delta, axis=0)
bias_1 += -alpha * numpy.meansum(layer_1_delta, axis=0)

NB - I have assumed you have fixed the gradient direction here.

Classification Loss Function

MLPClassifier is running a classifier using logloss, which is more efficient loss function that the mean squared error you are using, if your targets are class probabilities. You can use this too, simply by changing:

layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)

To

layer_2_delta = layer_2_error

This is the correct delta value to match the loss function

def calculateError(Y, Y_predicted):
    return -numpy.mean( (Y * numpy.log(Y_predicted) + 
       (1-Y)* numpy.log(1 - Y_predicted)))
  • but only when your output layer is sigmoid (the sigmoid derivative cancels out the derivatives of this log function).

    but only when your output layer is sigmoid (the sigmoid derivative cancels out the derivatives of this log function).

  • if you use this, you will want to reduce the learning rate for numerical stability. I suggest e.g. alpha = 0.01

Difference in Optimisers

You are running batched Stochastic Gradient Descent, which is the most basic optimiser type. The MLPClassifier is using Adam, which is faster and more robust. If you want to compete with it on even terms you need to implement a better optimiser. The simplest improvement is probably to add some momentum.

A note on gradient direction

As an aside,

layer_2_error = Y - layer_2

Should be

layer_2_error = layer_2 - Y

And all your update functions should be gradient descent e.g.

weights_2 += -alpha * layer_1.T.dot(layer_2_delta)

This makes no difference to the performance of your network, but I assume you have done this in the rest of the answer.

Scaling Input to Normalised Values

One clue to performance problems was in your first version which included the code:

X = preprocessing.scale(X)

With this included before training, then the inputs were scaled nicely for working with neural networks and the network converged quickly. Without this, then the network will still operate, but converges much more slowly. Increasing the max iterations to 1,000,000, then I get the result:

Goal reached after 209237 iterations: [ 0.00099997] is smaller than the goal of 0.001

You mention that you don't want to do scale the input, but really in general for NNs you should. It is worth looking at other differences though, because the lack of scaling does not prevent the MLPClassifier from converging.

Bias Gradients

This is one you spotted and mentioned in the comments. Your bias update is the same for each bias value. To correct this, you want something like:

# Update the bias    
bias_2 += -alpha * numpy.mean(layer_2_delta)
bias_1 += -alpha * numpy.mean(layer_1_delta)

NB - I have assumed you have fixed the gradient direction here.

Classification Loss Function

MLPClassifier is running a classifier using logloss, which is more efficient loss function that the mean squared error you are using, if your targets are class probabilities. You can use this too, simply by changing:

layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)

To

layer_2_delta = layer_2_error

This is the correct delta value to match the loss function

def calculateError(Y, Y_predicted):
    return -numpy.mean( (Y * numpy.log(Y_predicted) + 
       (1-Y)* numpy.log(1 - Y_predicted)))
  • but only when your output layer is sigmoid (the sigmoid derivative cancels out the derivatives of this log function).

Difference in Optimisers

You are running batched Stochastic Gradient Descent, which is the most basic optimiser type. The MLPClassifier is using Adam, which is faster and more robust. If you want to compete with it on even terms you need to implement a better optimiser. The simplest improvement is probably to add some momentum.

A note on gradient direction

As an aside,

layer_2_error = Y - layer_2

Should be

layer_2_error = layer_2 - Y

And all your update functions should be gradient descent e.g.

weights_2 += -alpha * layer_1.T.dot(layer_2_delta)

This makes no difference to the performance of your network, but I assume you have done this in the rest of the answer.

Scaling Input to Normalised Values

One clue to performance problems was in your first version which included the code:

X = preprocessing.scale(X)

With this included before training, then the inputs were scaled nicely for working with neural networks and the network converged quickly. Without this, then the network will still operate, but converges much more slowly. Increasing the max iterations to 1,000,000, then I get the result:

Goal reached after 209237 iterations: [ 0.00099997] is smaller than the goal of 0.001

You mention that you don't want to do scale the input, but really in general for NNs you should. It is worth looking at other differences though, because the lack of scaling does not prevent the MLPClassifier from converging.

Bias Gradients

This is one you spotted and mentioned in the comments. Your bias update is the same for each bias value. To correct this, you want something like:

# Update the bias    
bias_2 += -alpha * numpy.sum(layer_2_delta, axis=0)
bias_1 += -alpha * numpy.sum(layer_1_delta, axis=0)

NB - I have assumed you have fixed the gradient direction here.

Classification Loss Function

MLPClassifier is running a classifier using logloss, which is more efficient loss function that the mean squared error you are using, if your targets are class probabilities. You can use this too, simply by changing:

layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)

To

layer_2_delta = layer_2_error

This is the correct delta value to match the loss function

def calculateError(Y, Y_predicted):
    return -numpy.mean( (Y * numpy.log(Y_predicted) + 
       (1-Y)* numpy.log(1 - Y_predicted)))

  • but only when your output layer is sigmoid (the sigmoid derivative cancels out the derivatives of this log function).

  • if you use this, you will want to reduce the learning rate for numerical stability. I suggest e.g. alpha = 0.01

Difference in Optimisers

You are running batched Stochastic Gradient Descent, which is the most basic optimiser type. The MLPClassifier is using Adam, which is faster and more robust. If you want to compete with it on even terms you need to implement a better optimiser. The simplest improvement is probably to add some momentum.

Source Link
Neil Slater
Neil Slater
  • 29.2k
  • 5
  • 81
  • 100

A note on gradient direction

As an aside,

layer_2_error = Y - layer_2

Should be

layer_2_error = layer_2 - Y

And all your update functions should be gradient descent e.g.

weights_2 += -alpha * layer_1.T.dot(layer_2_delta)

This makes no difference to the performance of your network, but I assume you have done this in the rest of the answer.

Scaling Input to Normalised Values

One clue to performance problems was in your first version which included the code:

X = preprocessing.scale(X)

With this included before training, then the inputs were scaled nicely for working with neural networks and the network converged quickly. Without this, then the network will still operate, but converges much more slowly. Increasing the max iterations to 1,000,000, then I get the result:

Goal reached after 209237 iterations: [ 0.00099997] is smaller than the goal of 0.001

You mention that you don't want to do scale the input, but really in general for NNs you should. It is worth looking at other differences though, because the lack of scaling does not prevent the MLPClassifier from converging.

Bias Gradients

This is one you spotted and mentioned in the comments. Your bias update is the same for each bias value. To correct this, you want something like:

# Update the bias    
bias_2 += -alpha * numpy.mean(layer_2_delta)
bias_1 += -alpha * numpy.mean(layer_1_delta)

NB - I have assumed you have fixed the gradient direction here.

Classification Loss Function

MLPClassifier is running a classifier using logloss, which is more efficient loss function that the mean squared error you are using, if your targets are class probabilities. You can use this too, simply by changing:

layer_2_delta = layer_2_error*sigmoid_derivative(layer_2)

To

layer_2_delta = layer_2_error

This is the correct delta value to match the loss function

def calculateError(Y, Y_predicted):
    return -numpy.mean( (Y * numpy.log(Y_predicted) + 
       (1-Y)* numpy.log(1 - Y_predicted)))
  • but only when your output layer is sigmoid (the sigmoid derivative cancels out the derivatives of this log function).

Difference in Optimisers

You are running batched Stochastic Gradient Descent, which is the most basic optimiser type. The MLPClassifier is using Adam, which is faster and more robust. If you want to compete with it on even terms you need to implement a better optimiser. The simplest improvement is probably to add some momentum.