I am new to ML and this is my first Tensorflow project. I am doing regression with Neural Networks on a dataset with 17 features and 1 outcome. But for some reason my network is unable to follow the training data. I am getting massive errors in results, as can be seen from the plots below. I have also tried experimenting with different parameters (learning rate, nodes per layer, number of layers etc) but nothing seems to work. I have pasted the Tensorflow code here. I have also provided my cost and training plots as well as a link to my datasets. I'd be grateful if someone could please help me figure out what I am doing wrong. Thank you!

Links to the datasets- Features arranged in columns- https://drive.google.com/file/d/1U182Lhf67WygeSbv6BNEx5LHyL7Ba13O/view?usp=sharing Output column - https://drive.google.com/file/d/10XWo1d5mhIsxccQBgAyGDWDAVgu2BjAA/view?usp=sharing

enter image description here enter image description here

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf

# importing features and observations data for training and validation
training_filename_X = "training_set_X.csv"
training_filename_Y = "training_set_Y.csv"
validation_filename_X = "validation_set_X.csv"
test_filename_X = "test_set_X.csv"
test_filename_Y = "test_set_Y.csv"
validation_filename_Y = "validation_set_Y.csv"
training_features = np.loadtxt(training_filename_X, delimiter=',')
training_observations = np.loadtxt(training_filename_Y, delimiter=',')
validation_features = np.loadtxt(validation_filename_X, delimiter=',')
validation_observations = np.loadtxt(validation_filename_Y, delimiter=',')
test_features = np.loadtxt(test_filename_X, delimiter=',')
test_observations = np.loadtxt(test_filename_Y, delimiter=',')

# normalizing training data
training_features_stddev_arr = np.std(training_features, axis=0)
training_features_mean_arr = np.mean(training_features, axis=0)
normalized_training_features = (training_features-training_features_mean_arr)/training_features_stddev_arr

# layer parameters
n_nodes_hl1 = 5
n_nodes_hl2 = 5
n_nodes_hl3 = 3
no_features = 17
learning_rate = 0.001
epochs = 2000

cost_history = []

X = tf.placeholder(tf.float32)
Y = tf.placeholder(tf.float32)

# defining weights for each layer taken from a normal distribution with variance 2/n
hl1_weight = tf.Variable(tf.random_normal([no_features, n_nodes_hl1], stddev=np.sqrt(2/no_features)))
hl2_weight = tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2], stddev=np.sqrt(2/n_nodes_hl1)))
hl3_weight = tf.Variable(tf.random_normal([n_nodes_hl2, n_nodes_hl3], stddev=np.sqrt(2/n_nodes_hl2)))
output_weight = tf.Variable(tf.random_normal([n_nodes_hl3, 1], stddev=np.sqrt(2/n_nodes_hl3)))

# defining biases for each layer
hl1_bias = tf.Variable(tf.random_uniform([n_nodes_hl1], -1.0, 1.0))
hl2_bias = tf.Variable(tf.random_uniform([n_nodes_hl2], -1.0, 1.0))
hl3_bias = tf.Variable(tf.random_uniform([n_nodes_hl3], -1.0, 1.0))
output_bias = tf.Variable(tf.random_uniform([1], -1.0, 1.0))

# defining activation functions for each layer
hl1 = tf.sigmoid(tf.matmul(X, hl1_weight) + hl1_bias)
hl2 = tf.sigmoid(tf.matmul(hl1, hl2_weight) + hl2_bias)
hl3 = tf.sigmoid(tf.matmul(hl2, hl3_weight) + hl3_bias)
output = tf.matmul(hl3, output_weight) + output_bias

# using mean squared error cost function
cost  = tf.reduce_mean(tf.square(output - Y))

# using Gradient Descent algorithm
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)


init = tf.global_variables_initializer()

# running the network
with tf.Session() as sess:
    sess.run(init)

    for step in np.arange(epochs):
        sess.run(optimizer, feed_dict={X:normalized_training_features, Y:training_observations})
#        print (sess.run(cost, feed_dict={X:normalized_training_features, Y:training_observations}))
        cost_history.append(sess.run(cost,feed_dict={X:normalized_training_features, Y:training_observations}))

    pred_y = sess.run(output, feed_dict={X:normalized_training_features})

    plt.plot(range(len(cost_history)), cost_history)
  • Are they of the same scale? Also change your hidden layer neurons, your LR by *10 and epoch/10.. and report back.. – Aditya Jul 25 at 1:09
  • @Aditya I changed the LR and epoch (LR = 0.01, epoch = 200) as you suggested in the above code. It still gives the same plots for prediction. – user53799 Jul 25 at 2:40
  • @Aditya Also if you look at the data (google drive link provided), the features are of different scales but I have done feature scaling (normalizing training data) in the above code. Please let me know if the procedure I followed for normalization is not correct. Thanks! – user53799 Jul 25 at 2:43
  • 1
    Why in the world are you using squred error cost for sigmoid activation? – DuttaA Jul 25 at 4:48
  • @DuttaA Sorry, I am very new to the field of ML and my concepts aren't great right now. I thought the two could be used together. Also something similar was suggested here - stackoverflow.com/questions/34229140/… Please correct me if I am wrong. Thank you! – user53799 Jul 25 at 5:00

Ok I have found out atleast one thing why it's happening that way after asking my friends,

We can't use sigmoid as the last layer since it will always output something between 0 and 1 and hence that's why your predictions are not varying at all... i.e we can't at all use sigmoid as the last layer until it's a classification of let's say images and all...

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.