Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

242 questions
Filter by
Sorted by
Tagged with
18 views

### How can I regularize the output of a layer from scratch (without using Keras)?

I am trying to build a Convolutional Neural Network after reading notes from Stanford's cs231n course. I use ELU activation as activation function, and SoftMax as my classifier. Architecture is simple:...
27 views

### Why in this case are gradient steps not perpendicular to contour lines?

There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point. Why in this picture it seems that this rule is not respected? source: http://www....
6 views

### SGDClassifier partial_fit() for online learning - is one step of gradient descent enough?

I'm interested in incremental (online) learning for my logistic regression model trained with SGDClassifier. Basically updating the model as more labeled data comes ...
44 views

### Calculating the average of gradient decent

I am currently studying the backpropagation process and gradient decent algorithm form the book Neural Networks and Deep Learning written by Michael Nielsen and 3Blue1Brown channel in YouTube. My ...
18 views

### Cannot fig out error in my gradient function implementation in python

Im trying to implement following gradient descent function in Python for logistic regression: $∇θ(−logL)=−X^T 􏰀(y−e^{Xθ}􏰁)$ This is my python implementation: ...
19 views

39 views

### In an RNN, if the gradients don't vanish for long/distant terms, won't the derivative of the error be either divergent to infinity or oscillatory?

In chapter 6.1 of the book Deep Learning, the author tries to learn the XOR function by using a linear model (on page 168). Linear Model: $f(\mathbf{x};\mathbf{w},b)=\mathbf{x}^T\mathbf{w}+b$ MSE ...