2
$\begingroup$

am new to datascience and i want to learn linear regression so i coded linear regression from scratch and performed gradient descent to find the best $w_\theta$ and $b_\theta$ values using a tutorial. And it went just fine i was able to find the best $w_\theta$ , $b_\theta$ values and i ploted the line-of-best-fit (below).

Before Standization

And the gradient descent code i used to find the $w_\theta$ , $b_\theta$ values is below .

def step_gradient_descent(data,m,b,learning_rate=0.0001):
        b_gradient= m_gradient = 0
        N=float(len(data))
        for i in range(len(data)):
                [x,y]=data[i]
                y_=(m*x)+b
                m_gradient+= - (2/N)*(x*(y-y_))
                b_gradient+= -(2/N)*(y-y_)
        #print("m ={}, b ={}".format(m_gradient,b_gradient))
        m_new = m-(learning_rate*m_gradient)
        b_new = b-(learning_rate*b_gradient)
        return (m_new,b_new)

def perform_gradient_descent(data,m,b,lr=0.0001,epochs=1000):
        m_array=b_array=[]
        for i in range(epochs):
                if(i % 100 == 0 ):
                        print("Running {}/{}".format(i,epochs))
                (m,b) = step_gradient_descent(data,m,b,lr)
        return (m,b,m_array,b_array)

and then i performed features normalization / standization on $x$ , $y$ below

def featureNormalize(data):
    mean = np.mean(data , axis=0 )
    std = np.std(data , axis=0 )
    norm = ( data - mean ) / std
    return norm

and then after that i plotted the line-of-best-fit , which was different from the above plotted one. after feature normalization

So things i tried to get back the previous line-of-best-fit as before are

  • changing learning rate ( $r$ )
  • changing epochs ( $n$ )

which didnot work out. So my understanding is that feature normalization / standization should not have any effect on the gradient descent but that didnot happen in this case. Just curious to figure out what's happening in my case.

Link to my notebook

Thanks in Advance.

$\endgroup$

2 Answers 2

3
$\begingroup$

Standard normalization let gradient descent converge faster. That's why most cases normalization is applied (of course it also eliminate effect of different scale features values).

In this quora topic it is explained well.

Your learning rate is very small. I think there is not enough number of iterations to coverge local minimum. This graph explains well the effect of learning rate to gradient descent:

enter image description here

You may want to check my another answer for it.

$\endgroup$
8
  • $\begingroup$ makes sense let me try it $\endgroup$
    – guru_007
    Nov 30, 2019 at 16:14
  • 1
    $\begingroup$ Hope it works! Please share the results :) $\endgroup$ Nov 30, 2019 at 16:51
  • $\begingroup$ sure will so :-) $\endgroup$
    – guru_007
    Nov 30, 2019 at 18:29
  • 1
    $\begingroup$ I have found the answer: I just had to perform L2 Normalization with gradient-descent. please find the link to my notebook @llker Kurtulus $\endgroup$
    – guru_007
    Jan 15, 2020 at 18:27
  • 1
    $\begingroup$ @guru_007 One fun exercise is to create either a series of plots like your original for different epochs - or for a nice challenge append them into an animated image. You'll see the best fit line evolve over time. Also in general you should at least check that the mean of the residuals is zero - if not the algorithm hasn't converged. $\endgroup$ Jan 15, 2020 at 21:31
0
$\begingroup$

I just had to perform L2 Normalization , it worked like charm .

Theoritical Explanation :

$$ \begin{align} w_\theta &= - \frac{2}{N} \left( x * \left(y - \left(m*x+b\right) \right) \right) + \frac{\lambda}{2m} \sum_{i=1}^n \theta_i^2 \\ b_\theta &= - \frac{2}{N} \left(y - \left(m*x+b\right) \right) + \frac{\lambda}{2m} \sum_{i=1}^n \theta_i^2 \\ \end{align} $$

Change in code :

def step_gradient_descent_reg(data,m,b,learning_rate=0.0001,_lambda=1):
        b_gradient= m_gradient = 0
        N=float(len(data))
        for i in range(len(data)):
                [x,y]=data[i]
                y_=(m*x)+b
                m_gradient+= - ( (2/N)*(x*(y-y_)) ) + ( (_lambda / 2 * m ) * (np.sum(np.square(m_gradient))) )
                b_gradient+= - (2/N)*(y-y_)  + ( (_lambda / 2 * m ) * (np.sum(np.square(m_gradient))) )
        #print("m ={}, b ={}".format(m_gradient,b_gradient))
        m_new = m-(learning_rate*m_gradient)
        b_new = b-(learning_rate*b_gradient)
        return (m_new,b_new)

def perform_gradient_descent_reg(data,m,b,lr=0.0001,epochs=1000):
        m_array=b_array=[]
        for i in range(epochs):
                if(i % 100 == 0 ):
                        print("Running {}/{}".format(i,epochs))
                (m,b) = step_gradient_descent_reg(data,m,b,lr)
                m_array.append(m)
                b_array.append(b)
        #np.array(m_array),np.array(b_array)

        plot(data,m,b)

        return (m,b,m_array,b_array)

And as a result i got the correct result that i have wanted . line-of-best-fit with optimal $w_\theta$ and $b_\theta$ finally thanks to L2 Normalization technique.

enter image description here

$\endgroup$
1
  • $\begingroup$ L2 is Ridge Regularization technique - is used for Penalty, it works by shrinking estimates towards zero - "This increases the bias of the model, making the fit worse on the training data, but also decreases the variance". Normalization makes different sence - it is scaling to make matrix well conditioned for faster convergence of gradient descent $\endgroup$
    – JeeyCi
    Feb 15 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.