# Why does my linear regression model converge to a non-zero gradient value?

I have a basic 2D Linear Regression model coded out (using gradient descent), yet it doesn't seem to work as well as it should.

What I expect is that m and c should approach 4 and 3 respectively, and m's slope or c's slope should tend to 0; yet what is actually happening is that c's slope approaches a non-zero value, and c itself approaches a value depending on the epoch (around 0.5 with an epoch of 100.)

If I look at the graph of c, it very slowly tends up over time, though.

Code here:

import random, math
import matplotlib.pyplot as plt

def linreg(x, y):
""" Performs linear regression: input x, output y. """
n = float(len(x))
m = random.random()
c = random.random()
dm, dc = [], []
rate = 0.00001
epoch = 100
for run in range(epoch):
d_m = 0
d_c = 0
for i in range(len(x)):
d_m += (y[i] - m*x[i] - c)*x[i]
d_c += (y[i] - m*x[i] - c)
d_m *= -2/n
d_c *= -2/n
m -= d_m * rate
c -= d_c * rate
dm.append(d_m)
dc.append(d_c)
return m, c, dm, dc

x = [i for i in range(400)]
y = [4*i + 3 for i in x]

m, c, dm, dc = linreg(x, y)

print(m, c)

plt.grid()
plt.scatter(x, y)
plt.plot(x, [m*i + c for i in x], color='red')
plt.show()

plt.grid()
plt.plot([i for i in range(len(dm))], dm)
plt.plot([i for i in range(len(dc))], dc, color='red')
plt.show()


those x values are way larger than it should. If you normalize your x and increase the learning rate, it will converge easily. General assumption is that the input comes from a normal distribution so we transform the input.

Here is the code(notice that I only subtracted the mean from x and divided it by its standard deviation and I changed the rate to 0.05):

import random, math
import matplotlib.pyplot as plt
import numpy as np

def linreg(x, y):
""" Performs linear regression: input x, output y. """
n = float(len(x))
m = random.random()
c = random.random()
dm, dc = [], []
rate = 0.05
epoch = 100
for run in range(epoch):
d_m = 0
d_c = 0
for i in range(len(x)):
d_m += (y[i] - m*x[i] - c)*x[i]
d_c += (y[i] - m*x[i] - c)
d_m *= -2/n
d_c *= -2/n
m -= d_m * rate
c -= d_c * rate
dm.append(d_m)
dc.append(d_c)
return m, c, dm, dc

x = [i for i in range(400)]
x_mean = np.mean(x)
x_std = np.std(x)
x = [(i - x_mean) / x_std for i in x]
y = [4*i + 3 for i in x]

m, c, dm, dc = linreg(x, y)

print(m, c)

plt.grid()
plt.scatter(x, y)
plt.plot(x, [m*i + c for i in x], color='red')
plt.show()

plt.grid()
plt.plot([i for i in range(len(dm))], dm)
plt.plot([i for i in range(len(dc))], dc, color='red')
plt.show()


the output in my case:

3.999898446167822 2.999938454995635

• Yeah, but why does the input have to come from a normal distribution? That just seems weird. Shouldn't a perfectly linear approach like my approach work too? – virchau13 May 25 '19 at 12:38
• @virchau I'm sorry I can't provide you a thorough explanation on that. AFAIK, it's mostly convenient in calculation(calculation of log-likelihood is tractable assuming that the pdf is normal) and it yields robust models. – MGoksu May 28 '19 at 6:23