Linear Regression and learning rate

Could you please tell me why do we use a learning rate to move into the direction of the derivative to find the minimum? Why is it not good if you simply count it where is it 0?

• What do you mean with "Why is it not good if you simply count it where is it 0?"? – stmax Apr 18 '16 at 7:16

Learning rate gives the rate of speed where the gradient moves during gradient descent. Setting it too high would make your path instable, too low would make convergence slow. Put it to zero means your model isn't learning anything from the gradients.

• Hi, I meant, setting the derivative to zero. – user3435407 Apr 18 '16 at 6:46
• @user3435407 ??? Setting the derivative to zero means your model is not moving. – SmallChess Apr 18 '16 at 6:47
• I think what user3435407 means is setting the derivative of the loss function to zero and solving for the coefficients. That actually works for linear regression and gives the closed form solution β=(XTX)^−1 XTy. It only works for linear regression though - it does not work for logistic regression and most other generalized linear models. – stmax Apr 18 '16 at 7:26
• @user3435407 Is this what you mean? What stmax said? – SmallChess Apr 18 '16 at 7:27
• @stmax Thanks. If that's what he really means, I'll change my answer. – SmallChess Apr 18 '16 at 7:28

Use of learning can be understood using image below Usually cost function used in gradient descent are convex as should in image above. This will be similar also for data with multiple features because for such data we can reason this in similar way one feature at a time.

Let's say we are at point A during training at which point gradient is G, which means cost is increasing fastest in the direction in the direction of G. So we want to move in opposite direction of G with some step size which is learning rate.

As in fig above cost is increasing in direction of negative $$w_2$$ axis so we want to move in the direction of positive $$w_2$$ axis. But if we move too much in that direction i.e at point $$w_2=6$$ then actually cost value has increased. If we always move in same rate then we'll never reach the minimum point.

So we'll need a learning rate which is suitable for this cost function so that it is large enough that we'll have fast descent but low enough that it doesn't shoot other side of the curve

Let's take the function Y = x^2. Where is its minimum? It's where dy/dx = 0. Let's find that. dy/dx = 2x. But why 2x? Or simply how do we find a derivative function for any function just that easy? It's because we exactly know the relationship between Y and X. That is Y is always X*X. And same for all functions. Let's come to some data. Do we know the exact relationship between the input X and the target Y? No. Because we can't say 'Y is always "this" relationship with X' and that is actually what we r trying to find out. So what can we do ? Take small baby step in a way that our error is minimizing. Making big move might lead us to bad place.

So to answer ur question, in a case of Y = x^2, we know the relationship, so we find the minimum by formula not by analysis. But in case where the problem is mapping input data X to target Y, (where we don't know the relationship), we can't use formula to get minimum but analysis. U can use analysis for the former case, but it's little tedious I guess.