Problem with Linear Regression and Gradient Descent

Question

import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
import numpy as np

x = np.linspace(0, 10, 1000)
points = np.genfromtxt("Dokumente/Salary.csv", delimiter=",")
points2 = points[1:, :]

def gradient_descent(current_b, current_m, learning_rate, points):
    b_gradient = 0
    m_gradient = 0
    N = float(len(points))
    for i in range(0, len(points)):
        x = points[i, 0]
        y = points[i, 1] / (10 ** 4)
        b_gradient += -(2/N) * (y - ((x * current_m) + b_gradient))
        m_gradient += -(2/N) * x * (y - ((x * current_m) + current_b))
    b_new = current_b - (learning_rate * b_gradient)
    m_new = current_m - (learning_rate * m_gradient)
    return [b_new, m_new]

def compute_error(points, b, m):
    total_error = 0
    N = len(points)
    for i in range(0, len(points)):
        x = points[i, 0]
        y = points[i, 1] / (10 ** 4)
        total_error += (1/N) * ((y - ((m * x) + b)) ** 2)
    return total_error

def run(points, staring_b, starting_m, learning_rate, num_iterations):
    b = staring_b
    m = starting_m
    for i in range(num_iterations):
        learning_rate = learning_rate
        b, m = gradient_descent(b, m, learning_rate, np.array(points))
        plt.plot(i, compute_error(points, b, m), "ro")
    return [b, m]

learning_rate = 0.000001
init_b = 0
init_m = 0
number_of_iterations = 10000
print(compute_error(points2, init_b, init_m))
[b, m] = run(points2, init_b, init_m, learning_rate, number_of_iterations)
print(compute_error(points2, b, m))

plt.plot(points2[:,1] / (10 ** 4), "ro")
x = np.linspace(0, 35, 1000)
plt.plot(x, ((x * m) + b))
plt.show()

I played with learning rate and the number of iterations, but this does not fit the data intuively. Can i get some tips on my code please? Thank you in advance.

Answer 1

Gradient descent probably didn't converge. I'd sugest scaling your feature on a [-1,1] or [0,1] scales, try increasing your learning rate and maybe add some code which checks the gradient's slope and if there is perhaps sequence of <0,>0,<0 it means that the learning rate is locally too large and cost function can't converge. In that case, learning rate should be decreased.