2
$\begingroup$

am new to datascience and i want to learn linear regression so i coded linear regression from scratch and performed gradient descent to find the best $w_\theta$ and $b_\theta$ values using a tutorial. And it went just fine i was able to find the best $w_\theta$ , $b_\theta$ values and i ploted the line-of-best-fit (below).

Before Standization

And the gradient descent code i used to find the $w_\theta$ , $b_\theta$ values is below .

def step_gradient_descent(data,m,b,learning_rate=0.0001):
        b_gradient= m_gradient = 0
        N=float(len(data))
        for i in range(len(data)):
                [x,y]=data[i]
                y_=(m*x)+b
                m_gradient+= - (2/N)*(x*(y-y_))
                b_gradient+= -(2/N)*(y-y_)
        #print("m ={}, b ={}".format(m_gradient,b_gradient))
        m_new = m-(learning_rate*m_gradient)
        b_new = b-(learning_rate*b_gradient)
        return (m_new,b_new)

def perform_gradient_descent(data,m,b,lr=0.0001,epochs=1000):
        m_array=b_array=[]
        for i in range(epochs):
                if(i % 100 == 0 ):
                        print("Running {}/{}".format(i,epochs))
                (m,b) = step_gradient_descent(data,m,b,lr)
        return (m,b,m_array,b_array)

and then i performed features normalization / standization on $x$ , $y$ below

def featureNormalize(data):
    mean = np.mean(data , axis=0 )
    std = np.std(data , axis=0 )
    norm = ( data - mean ) / std
    return norm

and then after that i plotted the line-of-best-fit , which was different from the above plotted one. after feature normalization

So things i tried to get back the previous line-of-best-fit as before are

  • changing learning rate ( $r$ )
  • changing epochs ( $n$ )

which didnot work out. So my understanding is that feature normalization / standization should not have any effect on the gradient descent but that didnot happen in this case. Just curious to figure out what's happening in my case.

Link to my notebook

Thanks in Advance.

$\endgroup$
3
$\begingroup$

Standard normalization let gradient descent converge faster. That's why most cases normalization is applied (of course it also eliminate effect of different scale features values).

In this quora topic it is explained well.

Your learning rate is very small. I think there is not enough number of iterations to coverge local minimum. This graph explains well the effect of learning rate to gradient descent:

enter image description here

You may want to check my another answer for it.

| improve this answer | |
$\endgroup$
  • $\begingroup$ makes sense let me try it $\endgroup$ – guru_007 Nov 30 '19 at 16:14
  • 1
    $\begingroup$ Hope it works! Please share the results :) $\endgroup$ – Ilker Kurtulus Nov 30 '19 at 16:51
  • $\begingroup$ sure will so :-) $\endgroup$ – guru_007 Nov 30 '19 at 18:29
  • 1
    $\begingroup$ I have found the answer: I just had to perform L2 Normalization with gradient-descent. please find the link to my notebook @llker Kurtulus $\endgroup$ – guru_007 Jan 15 at 18:27
  • 1
    $\begingroup$ @guru_007 One fun exercise is to create either a series of plots like your original for different epochs - or for a nice challenge append them into an animated image. You'll see the best fit line evolve over time. Also in general you should at least check that the mean of the residuals is zero - if not the algorithm hasn't converged. $\endgroup$ – David Waterworth Jan 15 at 21:31
0
$\begingroup$

I just had to perform L2 Normalization , it worked like charm .

Theoritical Explanation :

$$ \begin{align} w_\theta &= - \frac{2}{N} \left( x * \left(y - \left(m*x+b\right) \right) \right) + \frac{\lambda}{2m} \sum_{i=1}^n \theta_i^2 \\ b_\theta &= - \frac{2}{N} \left(y - \left(m*x+b\right) \right) + \frac{\lambda}{2m} \sum_{i=1}^n \theta_i^2 \\ \end{align} $$

Change in code :

def step_gradient_descent_reg(data,m,b,learning_rate=0.0001,_lambda=1):
        b_gradient= m_gradient = 0
        N=float(len(data))
        for i in range(len(data)):
                [x,y]=data[i]
                y_=(m*x)+b
                m_gradient+= - ( (2/N)*(x*(y-y_)) ) + ( (_lambda / 2 * m ) * (np.sum(np.square(m_gradient))) )
                b_gradient+= - (2/N)*(y-y_)  + ( (_lambda / 2 * m ) * (np.sum(np.square(m_gradient))) )
        #print("m ={}, b ={}".format(m_gradient,b_gradient))
        m_new = m-(learning_rate*m_gradient)
        b_new = b-(learning_rate*b_gradient)
        return (m_new,b_new)

def perform_gradient_descent_reg(data,m,b,lr=0.0001,epochs=1000):
        m_array=b_array=[]
        for i in range(epochs):
                if(i % 100 == 0 ):
                        print("Running {}/{}".format(i,epochs))
                (m,b) = step_gradient_descent_reg(data,m,b,lr)
                m_array.append(m)
                b_array.append(b)
        #np.array(m_array),np.array(b_array)

        plot(data,m,b)

        return (m,b,m_array,b_array)

And as a result i got the correct result that i have wanted . line-of-best-fit with optimal $w_\theta$ and $b_\theta$ finally thanks to L2 Normalization technique.

enter image description here

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.