# Best way to normalize datasets for a linear regression model?

I am having some trouble getting a proper fit for a line using a simple linear regression model in tensorflow.

I've taken some housing data which I've normalized using the standard algorithm (dataset - mu) /sigma. The normalized feature data (independent variable) looks like this:

 [ 0.20502817 -0.16335143 -0.74031246  0.1520258  -0.54517939 -0.11878726
0.25697575 -1.00875229 -0.34951893  0.50774812 -0.07923326 -0.54728894
0.03679181  2.94664784 -0.67676237 -1.73918284  0.47214952 -0.34002597
0.67255646 -0.35980297  3.06610092 -0.99293069 -0.09531855 -0.50984448
1.90321329 -0.80307148 -1.29143155 -1.43382595 -0.58947987 -0.22479198
1.15353312  0.63247507 -0.64274593 -0.78382187  0.13435835  0.15149842
0.13963222  0.32817296 -0.2002685   1.47682115]


The normalized dependent variable data looks like this:

[ 0.45933219  0.00942554 -0.97430367  0.35768662 -0.13221173  1.2091766
-1.29030477 -1.04035663 -1.04035663  0.20938405 -0.67543235  0.04275196
3.44154673  0.6592907   1.36747709 -0.02390087 -0.64043961 -0.37382827
2.37560124 -0.22402602  0.04275196 -1.29030477 -0.69042924 -0.3921578
0.05941517 -1.54858451 -0.82373491 -1.50692649 -0.37382827 -0.45714431
0.80925958  1.10919735 -0.44464691  0.70928033  0.04275196 -0.05722729
-0.57378678 -0.10721692  0.87591242  0.90090723]


I'm using a standard tensorflow linear regression program to calculate the cost and the weights/biases to get a best fit line. The problem that I have is that when I plot this line, I can see visually that it isn't even close to a best fit, at least as far as I can see.

My original choice for feature normalization was to simply divide each x and y value by 1000 to get a smaller scale. This seems to give me a better fit but I'm not sure that this is an appropriate way to normalize features.

Here's what the data looks like before normalization.

Independent variable (House Lot Size)

[10603  9206  7018 10402  7758  9375 10800  6000  8500 11751  9525  7750
9965 21000  7259  3230 11616  8536 12376  8461 21453  6060  9464  7892
17043  6780  4928  4388  7590  8973 14200 12224  7388  6853 10335 10400
10355 11070  9066 15426]


Dependent Variable (House Sale Price)

[205000 178000 118964 198900 169500 250000 100000 115000 115000 190000
136900 180000 383970 217000 259500 176000 139000 155000 320000 163990
180000 100000 136000 153900 181000  84500 128000  87000 155000 150000
226000 244000 150750 220000 180000 174000 143000 171000 230000 231500]


Here is the code for calculating this linear regression model.

#!/usr/bin/python3

''' In this example, we're going to use linear regression in tensorflow to predict housing prices based
on the size of the lot as our features.
'''
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
import sys

# Normalize all of the features so that they're on the same numeric scale.
# Not doing this can lead to errors in the training process.
def normalize_features(dataset):
mu = np.mean(dataset,axis=0)
sigma = np.std(dataset,axis=0)
return (dataset - mu)/ sigma

rng = np.random

# learning_rate is the alpha value that we pass to the gradient descent algorithm.
learning_rate = 0.01
# How many cycles we're going to run to try and get our optimum fit.
training_epochs = 1000
display_step =  50

# We're going to pull in a the csv file and extract the X value (LotArea) and Y value (SalesPrice)
training_dataset = data_df[['LotArea','SalePrice']]

# We're going to do some feature scaling here and divide by 1000 for each X and Y.

train_X = training_dataset['LotArea'].values[:40] /1000
train_Y = training_dataset['SalePrice'].values[:40] /1000
#train_X = normalize_features(training_dataset['LotArea'].values[:40] )
#train_Y = normalize_features(training_dataset['SalePrice'].values[:40] )

#train_X = normalize_features(train_X)
#train_Y = normalize_features(train_Y)

# This is the total number of data samples that we're going to run through.
n_samples = train_X.shape

# Variable placeholders.
X = tf.placeholder('float')
Y = tf.placeholder('float')

W = tf.Variable(rng.randn(), name = 'weight')
b = tf.Variable(rng.randn(), name = 'bias')

# Here we describe our training model.  It's a linear regression model using the standard y = mx + b
# point slope formula. We calculate the cost by using least mean squares.

# This is our prediction algorithm: y = mx + b

# Let's now calculate the cost of the prediction algorithm using least mean squares
training_cost = tf.reduce_sum(tf.pow(prediction-Y,2))/(2*n_samples)
# This is our gradient descent optimizer algorithm.  We're passing in alpha, our learning rate
# and we want the minimum value of the training cost.

init = tf.global_variables_initializer()

# Now we'll run our training data through our model.
with tf.Session() as tf_session:

# Initialize all of our tensorflow variables.
tf_session.run(init)

# We'll run the data through for 1000 times (The value of training_epochs).

for epoch in range(training_epochs):

# For each training cycle, pass in the x and y values to our optimizer algorithm to calculate the cost.
for (x,y) in zip(train_X,train_Y):
tf_session.run(optimizer,feed_dict = {X: x, Y: y})

# For every fifty cycles, let's check and see how we're doing.
if (epoch + 1 ) % 50 == 0:
c = tf_session.run(training_cost,feed_dict = {X: train_X, Y: train_Y})
print ('Epoch: ', '%04d' % (epoch+1),'cost=','{:.9f}'.format(c), \
'W = ',tf_session.run(W), 'b = ',tf_session.run(b))

print ('Optimization finished')
print ('Training cost = ',training_cost,' W = ',tf_session.run(W), ' b  = ', tf_session.run(b),'\n')

plt.plot(train_X, train_Y, 'ro',label='Original data')
plt.axis((0,2,0,5))

plt.plot(train_X,tf_session.run(W) * train_X + tf_session.run(b), label = 'Fitted line')
plt.legend()
plt.show()

# We're now going to run test data to see how well our trained model works.
testing_dataset = data_df[['LotArea','SalePrice']]

test_X = testing_dataset['LotArea'].values/1000
test_Y = testing_dataset['SalePrice'].values/1000


If anybody could point me to what my flaws are in the understanding of how to normalize datasets correctly, I'd be very appreciative.

Thank you If you do this, your model will improve slightly. My $R^2$ improved from 0.1171 to 0.1873. While it's not a very strong linear relationship, you have some outliers pulled away the OLS fitting at the end of the x-scale. You might want to remove them or fit a robust regression (e.g. L1 cost function).