1
$\begingroup$

I have a basic 2D Linear Regression model coded out (using gradient descent), yet it doesn't seem to work as well as it should.

What I expect is that m and c should approach 4 and 3 respectively, and m's slope or c's slope should tend to 0; yet what is actually happening is that c's slope approaches a non-zero value, and c itself approaches a value depending on the epoch (around 0.5 with an epoch of 100.)

If I look at the graph of c, it very slowly tends up over time, though.

Code here:

import random, math
import matplotlib.pyplot as plt

def linreg(x, y):
    """ Performs linear regression: input x, output y. """
    n = float(len(x))
    m = random.random()
    c = random.random()
    dm, dc = [], []
    rate = 0.00001
    epoch = 100
    for run in range(epoch):
        d_m = 0
        d_c = 0
        for i in range(len(x)):
            d_m += (y[i] - m*x[i] - c)*x[i]
            d_c += (y[i] - m*x[i] - c)
        d_m *= -2/n
        d_c *= -2/n
        m -= d_m * rate
        c -= d_c * rate
        dm.append(d_m)
        dc.append(d_c)
    return m, c, dm, dc

x = [i for i in range(400)]
y = [4*i + 3 for i in x]


m, c, dm, dc = linreg(x, y)

print(m, c)

plt.grid()
plt.scatter(x, y)
plt.plot(x, [m*i + c for i in x], color='red')
plt.show()

plt.grid()
plt.plot([i for i in range(len(dm))], dm)
plt.plot([i for i in range(len(dc))], dc, color='red')
plt.show()
$\endgroup$
1
$\begingroup$

those x values are way larger than it should. If you normalize your x and increase the learning rate, it will converge easily. General assumption is that the input comes from a normal distribution so we transform the input.

Here is the code(notice that I only subtracted the mean from x and divided it by its standard deviation and I changed the rate to 0.05):

import random, math
import matplotlib.pyplot as plt
import numpy as np


def linreg(x, y):
    """ Performs linear regression: input x, output y. """
    n = float(len(x))
    m = random.random()
    c = random.random()
    dm, dc = [], []
    rate = 0.05
    epoch = 100
    for run in range(epoch):
        d_m = 0
        d_c = 0
        for i in range(len(x)):
            d_m += (y[i] - m*x[i] - c)*x[i]
            d_c += (y[i] - m*x[i] - c)
        d_m *= -2/n
        d_c *= -2/n
        m -= d_m * rate
        c -= d_c * rate
        dm.append(d_m)
        dc.append(d_c)
    return m, c, dm, dc


x = [i for i in range(400)]
x_mean = np.mean(x)
x_std = np.std(x)
x = [(i - x_mean) / x_std for i in x]
y = [4*i + 3 for i in x]

m, c, dm, dc = linreg(x, y)

print(m, c)

plt.grid()
plt.scatter(x, y)
plt.plot(x, [m*i + c for i in x], color='red')
plt.show()

plt.grid()
plt.plot([i for i in range(len(dm))], dm)
plt.plot([i for i in range(len(dc))], dc, color='red')
plt.show()

the output in my case:

3.999898446167822 2.999938454995635
$\endgroup$
  • $\begingroup$ Yeah, but why does the input have to come from a normal distribution? That just seems weird. Shouldn't a perfectly linear approach like my approach work too? $\endgroup$ – virchau13 May 25 '19 at 12:38
  • $\begingroup$ @virchau I'm sorry I can't provide you a thorough explanation on that. AFAIK, it's mostly convenient in calculation(calculation of log-likelihood is tractable assuming that the pdf is normal) and it yields robust models. $\endgroup$ – MGoksu May 28 '19 at 6:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.