SKNN regression problem

Question

I am trying to learn scikit-learn neuralnetwork and am coming up against the same problem in regression where no matter the dataset I getting a horizontal straight line for my fit.

here is an example using the Linear regression example from scikit-learn and then using the SKNN regressor , simple example code from the docs.

# -*- coding: utf-8 -*-

# Code source: Jaques Grobler
# http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html
# License: BSD 3 clause


import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model

# Load the diabetes dataset
diabetes = datasets.load_diabetes()


# Use only one feature
diabetes_X = diabetes.data[:, np.newaxis]
diabetes_X_temp = diabetes_X[:, :, 2]

# Split the data into training/testing sets
diabetes_X_train = diabetes_X_temp[:-20]
diabetes_X_test = diabetes_X_temp[-20:]

# Split the targets into training/testing sets
diabetes_y_train = diabetes.target[:-20]
diabetes_y_test = diabetes.target[-20:]

# Create linear regression object
regr = linear_model.LinearRegression()

# Train the model using the training sets
regr.fit(diabetes_X_train, diabetes_y_train)

print "Results of Linear Regression...."
print "================================\n"
# The coefficients
print('Coefficients: ', regr.coef_)
# The mean square error
print("Residual sum of squares: %.2f"
      % np.mean((regr.predict(diabetes_X_test) - diabetes_y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % regr.score(diabetes_X_test, diabetes_y_test))

# Plot outputs
plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')
plt.plot(diabetes_X_test, regr.predict(diabetes_X_test), color='blue',
         linewidth=3)

plt.xticks(())
plt.yticks(())

plt.show()

# Now using the sknn regressor
# http://scikit-neuralnetwork.readthedocs.org/en/latest/guide_beginners.html
#

from sknn.mlp import Regressor, Layer

nn = Regressor(
    layers=[
        Layer("Rectifier", units=200),
        Layer("Linear")],
    learning_rate=0.02,
    n_iter=10)

nn.fit(diabetes_X_train, diabetes_y_train)
print "Results of SKNN Regression...."
print "==============================\n"


# The coefficients
print('Coefficients: ', regr.coef_)
# The mean square error
print("Residual sum of squares: %.2f"
      % np.mean((nn.predict(diabetes_X_test) - diabetes_y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % nn.score(diabetes_X_test, diabetes_y_test))

# Plot outputs
plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')
plt.plot(diabetes_X_test, nn.predict(diabetes_X_test), color='blue',
         linewidth=3)

plt.xticks(())
plt.yticks(())

plt.show()

Results of Linear Regression:

('Coefficients: ', array([ 938.23786125]))
Residual sum of squares: 2548.07
Variance score: 0.47

 Results of SKNN Regression:

('Coefficients: ', array([ 938.23786125]))
Residual sum of squares: 5737.52
Variance score: -0.19

Changing the number of iterations to 1000 results in a score of -0.15

Answer 1

My best guess here is that your learning rate is way too high for the problem. You also probably have far more neurons in your hidden network than you need, seeing as you're using just one feature.

Recall that learning rate is controlling the "step size" in gradient descent and that for your dataset, it is likely far too high. I made some minor changes to your code and got better results than linear regression. Notice the use of 2 hidden neurons, a 0.001 learning rate, and 20 iterations.

# Now using the sknn regressor
# http://scikit-neuralnetwork.readthedocs.org/en/latest/guide_beginners.html


from sknn.mlp import Regressor, Layer

nn = Regressor(
    layers=[
        Layer("Rectifier", units=2),
        Layer("Linear")],
    learning_rate=0.001,
    n_iter=20)

nn.fit(diabetes_X_train, diabetes_y_train)
print("Results of SKNN Regression....")


# The coefficients
print('Coefficients: ', regr.coef_)
# The mean square error
print("Residual sum of squares: %.2f"
      % np.mean((nn.predict(diabetes_X_test) - diabetes_y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % nn.score(diabetes_X_test, diabetes_y_test))

# Plot outputs
plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')
plt.plot(diabetes_X_test, nn.predict(diabetes_X_test), color='blue',
         linewidth=3)

plt.xticks(())
plt.yticks(())

plt.show()

SKNN regression:

Results of SKNN Regression....
Coefficients:  [ 938.23786125]
Residual sum of squares: 6123.67
Variance score: 0.50