# Linear Regression and learning rate

Could you please tell me why do we use a learning rate to move into the direction of the derivative to find the minimum? Why is it not good if you simply count it where is it 0?

• What do you mean with "Why is it not good if you simply count it where is it 0?"? – stmax Apr 18 '16 at 7:16

Learning rate gives the rate of speed where the gradient moves during gradient descent. Setting it too high would make your path instable, too low would make convergence slow. Put it to zero means your model isn't learning anything from the gradients.

• Hi, I meant, setting the derivative to zero. – user3435407 Apr 18 '16 at 6:46
• @user3435407 ??? Setting the derivative to zero means your model is not moving. – SmallChess Apr 18 '16 at 6:47
• I think what user3435407 means is setting the derivative of the loss function to zero and solving for the coefficients. That actually works for linear regression and gives the closed form solution β=(XTX)^−1 XTy. It only works for linear regression though - it does not work for logistic regression and most other generalized linear models. – stmax Apr 18 '16 at 7:26
• @user3435407 Is this what you mean? What stmax said? – SmallChess Apr 18 '16 at 7:27
• @stmax Thanks. If that's what he really means, I'll change my answer. – SmallChess Apr 18 '16 at 7:28

Use of learning can be understood using image below Usually cost function used in gradient descent are convex as should in image above. This will be similar also for data with multiple features because for such data we can reason this in similar way one feature at a time.

Let's say we are at point A during training at which point gradient is G, which means cost is increasing fastest in the direction in the direction of G. So we want to move in opposite direction of G with some step size which is learning rate.

As in fig above cost is increasing in direction of negative $$w_2$$ axis so we want to move in the direction of positive $$w_2$$ axis. But if we move too much in that direction i.e at point $$w_2=6$$ then actually cost value has increased. If we always move in same rate then we'll never reach the minimum point.

So we'll need a learning rate which is suitable for this cost function so that it is large enough that we'll have fast descent but low enough that it doesn't shoot other side of the curve

The direction is governed by the derivative that we use in the Gradient Descent algorithm. Alpha basically tell how aggressive each step the algorithm makes. If you set alpha = 0.10 , it will take large steps in each iteration of GD than in the case of alpha = 0.01. In other words, alpha determine how large the changes in the parameter are made per iteration. Why is it not good if you simply count it where is it 0?

Setting alpha as zero will make the algorithm learn nothing from the examples.

How to learn alpha ?

It is hit and trail process. You try different values of alpha and plot the graph between the cost ( objective) function and number of iterations performed. Why is it not good if you simply count it where is it 0? Based on the above graph,aplha= 0.3 cause the GD algorithm to converge in less number of iterations.

Let's take the function Y = x^2. Where is its minimum? It's where dy/dx = 0. Let's find that. dy/dx = 2x. But why 2x? Or simply how do we find a derivative function for any function just that easy? It's because we exactly know the relationship between Y and X. That is Y is always X*X. And same for all functions. Let's come to some data. Do we know the exact relationship between the input X and the target Y? No. Because we can't say 'Y is always "this" relationship with X' and that is actually what we r trying to find out. So what can we do ? Take small baby step in a way that our error is minimizing. Making big move might lead us to bad place.

So to answer ur question, in a case of Y = x^2, we know the relationship, so we find the minimum by formula not by analysis. But in case where the problem is mapping input data X to target Y, (where we don't know the relationship), we can't use formula to get minimum but analysis. U can use analysis for the former case, but it's little tedious I guess.