# Does Feature Normalization affect Gradient Descent | Linear Regression

am new to datascience and i want to learn linear regression so i coded linear regression from scratch and performed gradient descent to find the best $$w_\theta$$ and $$b_\theta$$ values using a tutorial. And it went just fine i was able to find the best $$w_\theta$$ , $$b_\theta$$ values and i ploted the line-of-best-fit (below).

And the gradient descent code i used to find the $$w_\theta$$ , $$b_\theta$$ values is below .

def step_gradient_descent(data,m,b,learning_rate=0.0001):
N=float(len(data))
for i in range(len(data)):
[x,y]=data[i]
y_=(m*x)+b
return (m_new,b_new)

m_array=b_array=[]
for i in range(epochs):
if(i % 100 == 0 ):
print("Running {}/{}".format(i,epochs))
return (m,b,m_array,b_array)


and then i performed features normalization / standization on $$x$$ , $$y$$ below

def featureNormalize(data):
mean = np.mean(data , axis=0 )
std = np.std(data , axis=0 )
norm = ( data - mean ) / std
return norm


and then after that i plotted the line-of-best-fit , which was different from the above plotted one.

So things i tried to get back the previous line-of-best-fit as before are

• changing learning rate ( $$r$$ )
• changing epochs ( $$n$$ )

which didnot work out. So my understanding is that feature normalization / standization should not have any effect on the gradient descent but that didnot happen in this case. Just curious to figure out what's happening in my case.

Standard normalization let gradient descent converge faster. That's why most cases normalization is applied (of course it also eliminate effect of different scale features values).

In this quora topic it is explained well.

Your learning rate is very small. I think there is not enough number of iterations to coverge local minimum. This graph explains well the effect of learning rate to gradient descent:

You may want to check my another answer for it.

• makes sense let me try it – guru_007 Nov 30 '19 at 16:14
• Hope it works! Please share the results :) – Ilker Kurtulus Nov 30 '19 at 16:51
• sure will so :-) – guru_007 Nov 30 '19 at 18:29
• I have found the answer: I just had to perform L2 Normalization with gradient-descent. please find the link to my notebook @llker Kurtulus – guru_007 Jan 15 at 18:27
• @guru_007 One fun exercise is to create either a series of plots like your original for different epochs - or for a nice challenge append them into an animated image. You'll see the best fit line evolve over time. Also in general you should at least check that the mean of the residuals is zero - if not the algorithm hasn't converged. – David Waterworth Jan 15 at 21:31

I just had to perform L2 Normalization , it worked like charm .

### Theoritical Explanation :

\begin{align} w_\theta &= - \frac{2}{N} \left( x * \left(y - \left(m*x+b\right) \right) \right) + \frac{\lambda}{2m} \sum_{i=1}^n \theta_i^2 \\ b_\theta &= - \frac{2}{N} \left(y - \left(m*x+b\right) \right) + \frac{\lambda}{2m} \sum_{i=1}^n \theta_i^2 \\ \end{align}

### Change in code :

def step_gradient_descent_reg(data,m,b,learning_rate=0.0001,_lambda=1):
N=float(len(data))
for i in range(len(data)):
[x,y]=data[i]
y_=(m*x)+b
m_gradient+= - ( (2/N)*(x*(y-y_)) ) + ( (_lambda / 2 * m ) * (np.sum(np.square(m_gradient))) )
b_gradient+= - (2/N)*(y-y_)  + ( (_lambda / 2 * m ) * (np.sum(np.square(m_gradient))) )
return (m_new,b_new)

m_array=b_array=[]
for i in range(epochs):
if(i % 100 == 0 ):
print("Running {}/{}".format(i,epochs))

And as a result i got the correct result that i have wanted . line-of-best-fit with optimal $$w_\theta$$ and $$b_\theta$$ finally thanks to L2 Normalization technique.