0
$\begingroup$

Gradient descent in $n$ dimensions.

I'm learning about the downward gradient and the youtube videos and books only show a 2d curve as the slope drops to the minimum of the curve.

My question is, does the slope go down a multi-dimensional curve when the characteristics of a data set > 3?

How to see the slope going down in a data set with many characteristics?

$\endgroup$
  • $\begingroup$ You don't have to see it. You only have to trust the mathematical proofs. $\endgroup$ – Rodrigo de Azevedo Apr 9 at 8:58
0
$\begingroup$

Typically, you won’t use plots to show anything larger than 3D. Yes, gradient descent will always go down the curve of convex loss function, I.e, linear regression.

The gradient value is simply the derivative “slope” of the loss with respect to the inputs. It capture how “off” the output is on average for each feature. For example, if the error of an record is high and the feature is lower than optimal, the gradient will point in the direction that increase the feature’s value.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Multivariate gradient decent

Image courtesy : https://lh6.ggpht.com/-5RFMcz2bzWI/VA9NuUkB_vI/AAAAAAAAAdM/_DqqYO0nbX4/im2_thumb%25255B1%25255D.png?imgmax=800

My question is, does the slope go down a multi-dimensional curve when the characteristics of a data set > 3?

Ans : Yes, for any dimension the gradient decent job is to find direction to the lowest point (in line or plane). Then with learning rate, it will be decided how much the weight or bias should be adjusted.

How to see the slope going down in a data set with many characteristics?

Ans : For simplicity, if the validation accuracy is getting reduced by each epoch, then it is assumed going down.

Please check local minima and global minima concepts, which explains when there is more then one convergence point.

Does gradient descent always converge to an optimum?

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.