Could you please tell me why do we use a learning rate to move into the direction of the derivative to find the minimum? Why is it not good if you simply count it where is it 0?
Usually cost function used in gradient descent are convex as should in image above. This will be similar also for data with multiple features because for such data we can reason this in similar way one feature at a time.
Let's say we are at point A during training at which point gradient is G, which means cost is increasing fastest in the direction in the direction of G. So we want to move in opposite direction of G with some step size which is learning rate.
As in fig above cost is increasing in direction of negative $w_2$ axis so we want to move in the direction of positive $w_2$ axis. But if we move too much in that direction i.e at point $w_2=6$ then actually cost value has increased. If we always move in same rate then we'll never reach the minimum point.
So we'll need a learning rate which is suitable for this cost function so that it is large enough that we'll have fast descent but low enough that it doesn't shoot other side of the curve
The direction is governed by the derivative that we use in the Gradient Descent algorithm. Alpha basically tell how aggressive each step the algorithm makes. If you set alpha = 0.10 , it will take large steps in each iteration of GD than in the case of alpha = 0.01. In other words, alpha determine how large the changes in the parameter are made per iteration.
Gradient Descent Algorithm:
Why is it not good if you simply count it where is it 0?
Setting alpha as zero will make the algorithm learn nothing from the examples.
How to learn alpha ?
It is hit and trail process. You try different values of alpha and plot the graph between the cost ( objective) function and number of iterations performed. Why is it not good if you simply count it where is it 0?
Based on the above graph,aplha= 0.3 cause the GD algorithm to converge in less number of iterations.
Let's take the function Y = x^2. Where is its minimum? It's where dy/dx = 0. Let's find that. dy/dx = 2x. But why 2x? Or simply how do we find a derivative function for any function just that easy? It's because we exactly know the relationship between Y and X. That is Y is always X*X. And same for all functions. Let's come to some data. Do we know the exact relationship between the input X and the target Y? No. Because we can't say 'Y is always "this" relationship with X' and that is actually what we r trying to find out. So what can we do ? Take small baby step in a way that our error is minimizing. Making big move might lead us to bad place.
So to answer ur question, in a case of Y = x^2, we know the relationship, so we find the minimum by formula not by analysis. But in case where the problem is mapping input data X to target Y, (where we don't know the relationship), we can't use formula to get minimum but analysis. U can use analysis for the former case, but it's little tedious I guess.