Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

355 questions
Filter by
Sorted by
Tagged with
8k views

### How does Gradient Descent and Backpropagation work together?

Please forgive me as I am new to this. I have attached a diagram trying to model my understanding of neural network and Back-propagation? From videos on Coursera and resources online I formed the ...
110 views

### Training Examples used in Stochastic Gradient Descent

Hi I was reading the difference between GD and SGD and found the below link. [What is the difference between Gradient Descent and Stochastic Gradient Descent? Based on this information I wanted to ...
119 views

### Confusion on Delta Rule and Error

I'm currently reading Mitchell's book for Machine Learning, and he just started gradient descent. There's one part that's really confusing me. At one point, he gives this equation for the error of a ...
5k views

### How to get out of local minimums on stochastic gradient descent?

I'm not programming a neural network but I'm looking at it from a non-hands-on, theoretical point of view and I'm currently wondering how to escape a local minimum and how to get to a global minimum. ...
2k views

### Is empirical risk the same thing as loss function?

I am reading the article Stochastic Gradient Descent Tricks by Léon Bottou (avaible here) and on the very first page they introduce empirical risk $E_n(f) = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),y_i),$ ...
80 views

### Creating a convolutional layer with weight normalization?

This paper proposes parameterizing convolutional weights by having the primary weights normalized to unit norm plus an extra scalar weight: https://arxiv.org/pdf/1602.07868.pdf Implementations do ...
86 views

### How does a zero centered activation functions like tanh helps in gradient decent?

I know that, if X are all positive, or negative then the sign of the downstream gradient will be same as that of the upstream gradient, but what I don't understand is how the zero centered activation ...
202 views

### Intuition behind using the inverse of a Hessian matrix for automatically estimating the learning rate (aggression parameter) in gradient descent.

I am reviewing some course material where the lecturer suggests that instead of guessing the learning rate parameter in gradient descent implementation, one could use the inverse of the Hessian ...
164 views

### Is the magnitude of the gradient a weakness of Gradient Descent?

The formula for Gradient Descent is as follows: $$\mathbf{w} := \mathbf{w} - \alpha\; \triangledown C$$ The gradient itself points in the direction of steepest ascent, therefore it is logical to go ...
264 views

### What does the “randomly shuffle training samples” in stochastic gradient descent attain?

What does the "randomly shuffle training samples" in stochastic gradient descent attain? I interpreted that since the training samples are used to compute $$\hat{y}=f(w^t x)$$ so if the order of $x$...
139 views

### too few data while too many degrees of freedom in linear regression

To recognize handwritten digits, I have a fully connected network, containing only 2 layers: input layer (all pixels of the image) and output layer (0 or 1). I use the simplest linear regression for ...
999 views

### What is non-decomposable and/or non-differentiable loss function?

I have been reading some deep learning literature and came up with these concepts of non-decomposable and non-differentiable loss functions. My question is are these same thing? if not how are they ...
3k views

### Should the minimum value of a cost (loss) function be equal to zero?

We know optimization techniques search in the space of all the possible parameters for a parameter set that minimizes the cost function of the model. The most well-known loss functions, like MSE or ...
678 views

### Gradient Descent Python Implementation isnt converging

I'm trying to implement gradient descent in Python and following Andrew Ng course in order to follow the math. However, my implementation isn't working as I expected. It would be great if the ...
98 views

### How to Define a Cost Fucntion?

I want to define a cost function in python to identify optimum value in days when i should end a marketing campaign to save spend on campaigns not generating traffic good traffic. Problem is I dont ...
662 views

93 views

### Gradient descent multidimensional linear regression - does learning rate affects concurrency? [closed]

I wonder if gradient descent for multidimensional regression always finds the right result? I feel like this doesn't always have to be true. I have done some calculations and actually got correct ...
54k views

### What is the difference between Gradient Descent and Stochastic Gradient Descent?

What is the difference between Gradient Descent and Stochastic Gradient Descent? I am not very familiar with these, can you describe the difference with a short example?
218 views

### Algorithm for backpropagation through time

I am reading through this article trying to understand the bptt algorithm, in the context of an RNN. However there is one part I don’t understand: ...
29 views

### What is the best statistical measure tool to measure how close data is to fitted regression line if outliers are not fitted

I am using a custom algorithm based on Gradient descent which computes the best fit on a training dataset. In this data set I have outliers i.e. data points that I do not want to fit. The algorithm is ...
135 views

### What is the intuition behind Ridge Regression and Adapting Gradient Descent algorithms?

So I was going through Adaptive Gradient Descent, and learning the intuition behind it: optimizing the learning algorithm, and getting the model to converge faster. The way AdaGrad does this, is by ...
4k views

17 views

### Is there mathematical verification for TBPTT (truncated back propagation through time)

My question is in the title. I'm currently looking for a paper or academic reference to that algorithm, even in a novel framework. Thank you for all possible replies,
766 views

The Adam optimizer is often used for training neural networks; it typically avoids the need for hyperparameter search over parameters like the learning rate, etc. The Adam optimizer is an improvement ...
62 views

For adaptive learning optimizers such as Adam and RMSProb, the effective learning rate is not the same for all weight parameters. This means that we are not really following the direction of the ...
3k views

I have found mentions of two advantages in using gradients instead of actual residuals: 1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base ...
210 views

### Why Root Finding is important in Logistic Regression? (i.e. Newton Raphson)

I'd like to ask what is the main reason why we find the roots in logistic regression (i.e. why we use Newton Raphson method on logistic regression ). I understand the basics of Newton Raphson method, ...
130 views

### Why does Ensemble Averaging actually improve results?

Why does ensemble averaging work for neural networks? This is the main idea behind things like dropout. Consider an example of a hypersurface defined by the following image (white means lowest Cost). ...