Gradient descent in $n$ dimensions.
I'm learning about the downward gradient and the youtube videos and books only show a 2d curve as the slope drops to the minimum of the curve.
My question is, does the slope go down a multi-dimensional curve when the characteristics of a data set > 3?
How to see the slope going down in a data set with many characteristics?
Typically, you won’t use plots to show anything larger than 3D. Yes, gradient descent will always go down the curve of convex loss function, I.e, linear regression.
The gradient value is simply the derivative “slope” of the loss with respect to the inputs. It capture how “off” the output is on average for each feature. For example, if the error of an record is high and the feature is lower than optimal, the gradient will point in the direction that increase the feature’s value.
Image courtesy : https://lh6.ggpht.com/-5RFMcz2bzWI/VA9NuUkB_vI/AAAAAAAAAdM/_DqqYO0nbX4/im2_thumb%25255B1%25255D.png?imgmax=800
My question is, does the slope go down a multi-dimensional curve when the characteristics of a data set > 3?
Ans : Yes, for any dimension the gradient decent job is to find direction to the lowest point (in line or plane). Then with learning rate, it will be decided how much the weight or bias should be adjusted.
How to see the slope going down in a data set with many characteristics?
Ans : For simplicity, if the validation accuracy is getting reduced by each epoch, then it is assumed going down.
Please check local minima and global minima concepts, which explains when there is more then one convergence point.
