Does Feature Normalization affect Gradient Descent | Linear Regression

Question

am new to datascience and i want to learn linear regression so i coded linear regression from scratch and performed gradient descent to find the best $w_\theta$ and $b_\theta$ values using a tutorial. And it went just fine i was able to find the best $w_\theta$ , $b_\theta$ values and i ploted the line-of-best-fit (below).

And the gradient descent code i used to find the $w_\theta$ , $b_\theta$ values is below .

def step_gradient_descent(data,m,b,learning_rate=0.0001):
        b_gradient= m_gradient = 0
        N=float(len(data))
        for i in range(len(data)):
                [x,y]=data[i]
                y_=(m*x)+b
                m_gradient+= - (2/N)*(x*(y-y_))
                b_gradient+= -(2/N)*(y-y_)
        #print("m ={}, b ={}".format(m_gradient,b_gradient))
        m_new = m-(learning_rate*m_gradient)
        b_new = b-(learning_rate*b_gradient)
        return (m_new,b_new)

def perform_gradient_descent(data,m,b,lr=0.0001,epochs=1000):
        m_array=b_array=[]
        for i in range(epochs):
                if(i % 100 == 0 ):
                        print("Running {}/{}".format(i,epochs))
                (m,b) = step_gradient_descent(data,m,b,lr)
        return (m,b,m_array,b_array)

and then i performed features normalization / standization on $x$ , $y$ below

def featureNormalize(data):
    mean = np.mean(data , axis=0 )
    std = np.std(data , axis=0 )
    norm = ( data - mean ) / std
    return norm

and then after that i plotted the line-of-best-fit , which was different from the above plotted one.

So things i tried to get back the previous line-of-best-fit as before are

• changing learning rate ( $r$ )
• changing epochs ( $n$ )

which didnot work out. So my understanding is that feature normalization / standization should not have any effect on the gradient descent but that didnot happen in this case. Just curious to figure out what's happening in my case.

Link to my notebook

Thanks in Advance.

Answer 1

Standard normalization let gradient descent converge faster. That's why most cases normalization is applied (of course it also eliminate effect of different scale features values).

In this quora topic it is explained well.

Your learning rate is very small. I think there is not enough number of iterations to coverge local minimum. This graph explains well the effect of learning rate to gradient descent:

You may want to check my another answer for it.

Answer 2

I just had to perform L2 Normalization , it worked like charm .

Theoritical Explanation :

$$ \begin{align} w_\theta &= - \frac{2}{N} \left( x * \left(y - \left(m*x+b\right) \right) \right) + \frac{\lambda}{2m} \sum_{i=1}^n \theta_i^2 \\ b_\theta &= - \frac{2}{N} \left(y - \left(m*x+b\right) \right) + \frac{\lambda}{2m} \sum_{i=1}^n \theta_i^2 \\ \end{align} $$

Change in code :

def step_gradient_descent_reg(data,m,b,learning_rate=0.0001,_lambda=1):
        b_gradient= m_gradient = 0
        N=float(len(data))
        for i in range(len(data)):
                [x,y]=data[i]
                y_=(m*x)+b
                m_gradient+= - ( (2/N)*(x*(y-y_)) ) + ( (_lambda / 2 * m ) * (np.sum(np.square(m_gradient))) )
                b_gradient+= - (2/N)*(y-y_)  + ( (_lambda / 2 * m ) * (np.sum(np.square(m_gradient))) )
        #print("m ={}, b ={}".format(m_gradient,b_gradient))
        m_new = m-(learning_rate*m_gradient)
        b_new = b-(learning_rate*b_gradient)
        return (m_new,b_new)

def perform_gradient_descent_reg(data,m,b,lr=0.0001,epochs=1000):
        m_array=b_array=[]
        for i in range(epochs):
                if(i % 100 == 0 ):
                        print("Running {}/{}".format(i,epochs))
                (m,b) = step_gradient_descent_reg(data,m,b,lr)
                m_array.append(m)
                b_array.append(b)
        #np.array(m_array),np.array(b_array)

        plot(data,m,b)

        return (m,b,m_array,b_array)

And as a result i got the correct result that i have wanted . line-of-best-fit with optimal $w_\theta$ and $b_\theta$ finally thanks to L2 Normalization technique.