Tensorflow regression model giving same prediction every time

Question

import tensorflow as tf

x = tf.placeholder(tf.float32, [None,4])    # input vector    
w1 = tf.Variable(tf.random_normal([4,2]))   # weights between first and second layers
b1 = tf.Variable(tf.zeros([2]))             # biases added to hidden layer
w2 = tf.Variable(tf.random_normal([2,1]))   # weights between second and third layer
b2 = tf.Variable(tf.zeros([1]))             # biases added to third (output) layer

def feedForward(x,w,b):                     # function for forward propagation
    Input = tf.add(tf.matmul(x,w), b)
    Output = tf.sigmoid(Input)
    return Output

Out1 = feedForward(x,w1,b1)                # output of first layer
Out2 = feedForward(Out1,w2,b2)             # output of second layer
MHat = 50*Out2                             # final prediction is in the range (0,50)

M = tf.placeholder(tf.float32, [None,1])   # placeholder for actual (target value of marks)
J = tf.reduce_mean(tf.square(MHat - M))    # cost function -- mean square errors                          
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(J)     # minimize J using Gradient Descent
sess = tf.InteractiveSession()             # create interactive session 

tf.global_variables_initializer().run()    # initialize all weight and bias variables with specified values
xs = [[1,3,9,7],    
      [7,9,8,2],                           # x training data
      [2,4,6,5]]

Ms = [[47],
      [43],                                # M training data
      [39]]

for _ in range(1000):                      # performing learning process on training data 1000 times
    sess.run(train_step, feed_dict = {x:xs, M:Ms})

>>> print(sess.run(MHat, feed_dict = {x:[[1,15,9,7]]}))
[[50.]]

>>> print(sess.run(MHat, feed_dict = {x:[[3,8,1,2]]}))
[[50.]]

>>> print(sess.run(MHat, feed_dict = {x:[[6,7,10,9]]}))
[[50.]]

In this code, I am trying to predict the marks M obtained by a student in a test out of 50 given how many hours he/she slept, studied, used electronics and played the day before the test. These 4 features come under the input feature vector x.

To solve this regression problem, I am using a deep neural network with an input layer with 4 perceptrons (the input features), a hidden layer with two perceptrons and an output layer with one perceptron.

I have used sigmoid as the activation function. But, I am getting the exact same prediction([[50.0]]) for M for all possible input vectors I feed in.

Can someone please tell me what is wrong with the code above, and why I get the same result each time?

Answer 1

Your network design/logic is basically correct, but you are seeing some very common problems with neural network numerical stability. This results in your weights diverging and not training accurately.

Here are the fixes, any one of them might help a little, but the first two should be used for nearly all neural network projects.

1. Inputs need to be scaled to work with neural networks.

This is called input normalisation. Usually you would do this in data preprocessing, but for your simple network we can include the scaling at the input:

x_normalised = x * 0.2 - 0.5             # Arbitrary scaling I just made up
Out1 = feedForward(x_normalised,w1,b1)   # output of first layer

The most common scaling operation is to take all the training data and convert it so that it has mean $0$ and standard deviation $1$ (typically per feature, as opposed to global scaling for all data as I have done here) - store the values used to achieve that and apply them to all following data for training, tests etc.

2. Adjust learning rate until you get a working value.

train_step = tf.train.GradientDescentOptimizer(0.001).minimize(J) 

A value that is too high will cause your training to fail. A value that is to low will take ages to learn anything.

3. For small training sets, use more iterations than you might think

This is not a general thing, but specific to demos with tiny amounts of training data like your example, or the commonly-used "learning XOR function".

for _ in range(10000):                      # performing learning process

   sess.run(train_step, feed_dict = {x:xs, M:Ms})

With your very simple network actually this may cause over-fitting to the training data, so you will have to play with a value that gives you "sensible" results. However, in general how to spot and minimise over-fitting is a whole broad subject in itself, based on how you test and measure generalisation. This will need other questions if you are not sure when you learn it. It should be high on your list of things to learn though . . . it is a critical skill in producing useful neural networks that solve real problems.

Answer 2

In addition to Neil Slater's answer above, using ReLU to activate your neurons will converge it faster (allows you to train on a shorter period of time).

def feedForward(x,w,b):
    Input = tf.add(tf.matmul(x,w), b)
    Output = tf.nn.relu(Input)           # this ReLU
    return Output

MHat = tf.nn.relu(Out2)                  # and for the prediction

Sigmoid is mostly used for values ranging 0.0 to 1.0 (see this).