I have recently started learning Neural networks and Python. I am trying out linear regression for a dataset with 14 features and 1 outcome. I have divided the data into training and test data. I have experimented with many parameters (learning rate, nodes per layer, number of layers, number of steps and optimization algorithm) but my test errors are as high as 150%.

I have posted my code below along with the cost curve (cost vs epochs). Where am making a mistake and what should I change? Or can you suggest some other important checks?

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf

# importing features and observations data for training and validation
training_filename_X = "training_set_X.csv"
training_filename_Y = "training_set_Y.csv"
test_filename_X = "test_set_X.csv"
test_filename_Y = "test_set_Y.csv"
training_features = np.loadtxt(training_filename_X, delimiter=',')
training_observations = np.loadtxt(training_filename_Y, delimiter=',')
test_features = np.loadtxt(test_filename_X, delimiter=',')
test_observations = np.loadtxt(test_filename_Y, delimiter=',')

# normalizing training data
training_features_stddev_arr = np.std(training_features, axis=0)
training_features_mean_arr = np.mean(training_features, axis=0)
normalized_training_features = (training_features-training_features_mean_arr)/training_features_stddev_arr

# normalizing validation data with training set mean and standard deviation
normalized_validation_features = (validation_features-training_features_mean_arr)/training_features_stddev_arr

# normalizing test data with training set mean and standard deviation
normalized_test_features = (test_features-training_features_mean_arr)/training_features_stddev_arr

# layer parameters
n_nodes_hl1 = 20
n_nodes_hl2 = 20
n_nodes_hl3 = 20
no_features = 14
learning_rate = 0.01
epochs = 200

cost_history = np.empty(shape=[1], dtype=float)

X = tf.placeholder(tf.float32)
Y = tf.placeholder(tf.float32)

# defining weights for each layer taken from a normal distribution with variance 2/n
hl1_weight = tf.Variable(tf.random_normal([no_features, n_nodes_hl1], stddev=np.sqrt(2/no_features)))
hl2_weight = tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2], stddev=np.sqrt(2/n_nodes_hl1)))
hl3_weight = tf.Variable(tf.random_normal([n_nodes_hl2, n_nodes_hl3], stddev=np.sqrt(2/n_nodes_hl2)))
output_weight = tf.Variable(tf.random_normal([n_nodes_hl3, 1], stddev=np.sqrt(2/n_nodes_hl3)))

# defining biases for each layer
hl1_bias = tf.Variable(tf.random_uniform([n_nodes_hl1], -1.0, 1.0))
hl2_bias = tf.Variable(tf.random_uniform([n_nodes_hl2], -1.0, 1.0))
hl3_bias = tf.Variable(tf.random_uniform([n_nodes_hl3], -1.0, 1.0))
output_bias = tf.Variable(tf.random_uniform([1], -1.0, 1.0))

# defining activation functions for each layer
hl1 = tf.sigmoid(tf.matmul(X, hl1_weight) + hl1_bias)
hl2 = tf.sigmoid(tf.matmul(hl1, hl2_weight) + hl2_bias)
hl3 = tf.sigmoid(tf.matmul(hl2, hl3_weight) + hl3_bias)
output = tf.matmul(hl3, output_weight) + output_bias

# using mean squared error cost function
cost  = tf.reduce_mean(tf.square(output - Y))

# using Gradient Descent algorithm
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

init = tf.global_variables_initializer()

# running the network
with tf.Session() as sess:

    for step in np.arange(epochs):
        sess.run(optimizer, feed_dict={X:normalized_training_features, Y:training_observations})
        print (sess.run(cost, feed_dict={X:normalized_training_features, Y:training_observations}))
        cost_history = np.append(cost_history, sess.run(cost,feed_dict={X:normalized_training_features, Y:training_observations}))

    pred_y = sess.run(output, feed_dict={X:normalized_test_features})
    print (sess.run(output, feed_dict={X:normalized_test_features}))
    mse = tf.reduce_mean(tf.square(pred_y - test_observations))
    print("MSE: %4f" % sess.run(mse))

# plotting the cost history
plt.plot(range(len(cost_history)), cost_history)
plt.axis([0, epochs, 0, np.max(cost_history)])

enter image description here

  • $\begingroup$ How many instances do you have in your dataset? And can you plot the predicted vs the actual? $\endgroup$ – JahKnows Jun 19 '18 at 2:07
  • $\begingroup$ @JahKnows I have 900 instances in my data set and 169 in my test set. I have 14 features. I don't know if they are sufficient. What is your opinion? $\endgroup$ – user53799 Jun 20 '18 at 5:11

Based on my observations

  • I don't see your training_observations (output) normalized. Refer this example
  • I see you are using sigmoid in all your hidden layers. Generally tanh is used as mean of data is closer to zero, this actually makes the learning for next layer a bit easier.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.