# SKNN regression problem

I am trying to learn scikit-learn neuralnetwork and am coming up against the same problem in regression where no matter the dataset I getting a horizontal straight line for my fit.

here is an example using the Linear regression example from scikit-learn and then using the SKNN regressor , simple example code from the docs.

# -*- coding: utf-8 -*-

# Code source: Jaques Grobler
# http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model

# Use only one feature
diabetes_X = diabetes.data[:, np.newaxis]
diabetes_X_temp = diabetes_X[:, :, 2]

# Split the data into training/testing sets
diabetes_X_train = diabetes_X_temp[:-20]
diabetes_X_test = diabetes_X_temp[-20:]

# Split the targets into training/testing sets
diabetes_y_train = diabetes.target[:-20]
diabetes_y_test = diabetes.target[-20:]

# Create linear regression object
regr = linear_model.LinearRegression()

# Train the model using the training sets
regr.fit(diabetes_X_train, diabetes_y_train)

print "Results of Linear Regression...."
print "================================\n"
# The coefficients
print('Coefficients: ', regr.coef_)
# The mean square error
print("Residual sum of squares: %.2f"
% np.mean((regr.predict(diabetes_X_test) - diabetes_y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % regr.score(diabetes_X_test, diabetes_y_test))

# Plot outputs
plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')
plt.plot(diabetes_X_test, regr.predict(diabetes_X_test), color='blue',
linewidth=3)

plt.xticks(())
plt.yticks(())

plt.show()

# Now using the sknn regressor
#

from sknn.mlp import Regressor, Layer

nn = Regressor(
layers=[
Layer("Rectifier", units=200),
Layer("Linear")],
learning_rate=0.02,
n_iter=10)

nn.fit(diabetes_X_train, diabetes_y_train)
print "Results of SKNN Regression...."
print "==============================\n"

# The coefficients
print('Coefficients: ', regr.coef_)
# The mean square error
print("Residual sum of squares: %.2f"
% np.mean((nn.predict(diabetes_X_test) - diabetes_y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % nn.score(diabetes_X_test, diabetes_y_test))

# Plot outputs
plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')
plt.plot(diabetes_X_test, nn.predict(diabetes_X_test), color='blue',
linewidth=3)

plt.xticks(())
plt.yticks(())

plt.show()



Results of Linear Regression:

('Coefficients: ', array([ 938.23786125]))
Residual sum of squares: 2548.07
Variance score: 0.47


￼ Results of SKNN Regression:

('Coefficients: ', array([ 938.23786125]))
Residual sum of squares: 5737.52
Variance score: -0.19


Changing the number of iterations to 1000 results in a score of -0.15

My best guess here is that your learning rate is way too high for the problem. You also probably have far more neurons in your hidden network than you need, seeing as you're using just one feature.

Recall that learning rate is controlling the "step size" in gradient descent and that for your dataset, it is likely far too high. I made some minor changes to your code and got better results than linear regression. Notice the use of 2 hidden neurons, a 0.001 learning rate, and 20 iterations.

# Now using the sknn regressor

from sknn.mlp import Regressor, Layer

nn = Regressor(
layers=[
Layer("Rectifier", units=2),
Layer("Linear")],
learning_rate=0.001,
n_iter=20)

nn.fit(diabetes_X_train, diabetes_y_train)
print("Results of SKNN Regression....")

# The coefficients
print('Coefficients: ', regr.coef_)
# The mean square error
print("Residual sum of squares: %.2f"
% np.mean((nn.predict(diabetes_X_test) - diabetes_y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % nn.score(diabetes_X_test, diabetes_y_test))

# Plot outputs
plt.scatter(diabetes_X_test, diabetes_y_test,  color='black')
plt.plot(diabetes_X_test, nn.predict(diabetes_X_test), color='blue',
linewidth=3)

plt.xticks(())
plt.yticks(())

plt.show()


SKNN regression:

Results of SKNN Regression....
Coefficients:  [ 938.23786125]
Residual sum of squares: 6123.67
Variance score: 0.50

• Thanks. Can i ask can you offer any guidance as a good rule if thumb for the number of hidden neurons and learning rate as a first approximation? Given a set number of features? Eg something like sqrt number features for neurons? Commented Sep 20, 2015 at 16:11
• @Chris - sorry for the delay. There isn't some idealized rule for this, though some general guidelines exist. This is the best example i've found for succinct rules to apply when building NNs and this paper by geoffry hinton gives a much more complete view on the matter. Commented Sep 23, 2015 at 14:31