# Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

405 questions
Filter by
Sorted by
Tagged with
42 views
+50

### Verifying my understanding of MLE & Gradient Descent in Logistic Regression

Here is my understanding of the relation between MLE & Gradient Descent in Logistic Regression. Please correct me if I'm wrong: 1) MLE estimates optimal parameters by taking the partial derivative ...
• 315
11 views

### How do you find the eigenvalues of the matrix for the following momentum gradient descent?

The following question is based purely on the material available on MIT's open courseware youtube channel. (https://www.youtube.com/watch?v=wrEcHhoJxjM). In it, Professor Gilbert Strang explains the ...
20 views

### ResNet: Derive the gradient matrices w.r.t. W1 and W2 and backprop equation in a Residual Network

How would I go about step by step deriving stochastic gradient matrices w.r.t. W1 and W2 and backpropagation equation in a residual block that is a part of a larger ResNet network with forward ...
13 views

### From what function do come the gradients that I use to adjust weights?

I have a question about the loss function and the gradient. So I'm following the fastai (https://github.com/fastai/fastbook) course and at the end of 4th chapter, I got myself wondering. From what ...
28 views

### Difference between OLS and Gradient Descent in Linear Regression

I understand what Ordinary Least Squares and Gradient Descent do but I am just confused about the difference between them. The only difference I can think of are- Gradient Descent is iterative while ...
• 121
1 vote
22 views

### Need help to understand the formula of gradient descent with multiple features

I am trying to implement gradient descent with multiple features after listening to Andrew Ng's Coursera lecture. gradient descent for multiple features So for example when calculating for theta 1, ...
• 11
5 views

### What is "Gradient × Hidden States" explainability method? Is there any documentation about it?

I am doing a literature review on post-hoc explainability methods based on gradient. I stumbled upon one I didn't heard of to extract highlights from a trained model in this post-hoc fashion: We ...
1 vote
7 views

• 165
16 views

### RMSprop in weight update - what if vertical slopes small and horizontal slopes large?

I have a question regarding the intuition behind RMSprop, As shown in the lecture video of Deep Learning Specialization by Andrew Ng, RMSprop helps to reduce the oscillation (the values of the ...
1 vote
33 views

### Is saddle point a cause for the vanishing gradient problem

I am a beginner to neural networks and I am writing a report summarising on the causes and solutions to the vanishing gradient problem. From what I have read, the 2 main causes are the repeated ...
• 113
11 views

### cost function diverging in batch gradient descent

I am trying to implement the gradient descent method in python. I would like the calculation to stop when abs(J-J_new) reaches a certain tolerance level (i.e. it ...
32 views

### Gradient descent in linear regression converges but the trend line is incorrect

For the dataset https://physics.info/linear-regression/dash-world.txt, I have been trying to implement linear regression for predicting the men record times as a function of year. I have used gradient ...
19 views

116 views

### Why the gradient of a ReLU for X>0 is 1?

Gradient is derivative of several variables. I can't understand why is the gradient of a ReLU for X>0 is 1 ? and 0 for x < 0 ? I tried to search for proof and examples but didn't found any good ...
• 267
16 views

### Step size finds by quadratic fitting in steepest descent

I have a function $f=(1-x_1)^2 + (x_2-(x_1^2))^2$ and initial point $[0,5]$. I wonder how I will find step size by quadratic fitting using the (e.g. $0.01$) value in Steepest Descent with Matlab. To ...
• 1