Questions tagged [gradient-descent]

Gradient Descent is an algorithm for finding the minimum of a function. It iteratively calculates partial derivatives (gradients) of the function and descends in steps proportional to those partial derivatives. One major application of Gradient Descent is fitting a parameterized model to a set of data: the function to be minimized is an error function for the model.

Filter by
Sorted by
Tagged with
39
votes
5answers
45k views

What is the difference between Gradient Descent and Stochastic Gradient Descent?

What is the difference between Gradient Descent and Stochastic Gradient Descent? I am not very familiar with these, can you describe the difference with a short example?
31
votes
5answers
31k views

Does gradient descent always converge to an optimum?

I am wondering whether there is any scenario in which gradient descent does not converge to a minimum. I am aware that gradient descent is not always guaranteed to converge to a global optimum. I am ...
27
votes
4answers
37k views

Scikit-learn: Getting SGDClassifier to predict as well as a Logistic Regression

A way to train a Logistic Regression is by using stochastic gradient descent, which scikit-learn offers an interface to. What I would like to do is take a scikit-learn's SGDClassifier and have it ...
17
votes
1answer
24k views

Why ReLU is better than the other activation functions

Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
13
votes
4answers
3k views

Is Gradient Descent central to every optimizer?

I want to know whether Gradient descent is the main algorithm used in optimizers like Adam, Adagrad, RMSProp and several other optimizers.
11
votes
2answers
8k views

Why is learning rate causing my neural network's weights to skyrocket?

I am using tensorflow to write simple neural networks for a bit of research and I have had many problems with 'nan' weights while training. I tried many different solutions like changing the optimizer,...
10
votes
2answers
8k views

Stochastic gradient descent based on vector operations?

let's assume that I want to train a stochastic gradient descent regression algorithm using a dataset that has N samples. Since the size of the dataset is fixed, I will reuse the data T times. At each ...
10
votes
1answer
370 views

How flexible is the link between objective function and output layer activation function?

It seems standard in many neural network packages to pair up the objective function to be minimised with the activation function in the output layer. For instance, for a linear output layer used for ...
10
votes
4answers
965 views

Why does it speed up gradient descent if the function is smooth?

I now read a book titled "Hands-on Machine Learning with Scikit-Learn and TensorFlow" and on the chapter 11, it has the following description on the explanation of ELU (Exponential ReLU). Third, ...
9
votes
1answer
4k views

Understanding dropout and gradient descent

I am looking at how to implement dropout on deep neural network, and I found something counter intuitive. In the forward phase dropout mask activations with a random tensor of 1s and 0s to force net ...
9
votes
3answers
417 views

Why is taking the gradient of the average error in SGD not correct, but rather the average of the gradients of single errors?

I am a little confused about taking averages in cost functions and SGD. So far I always thought in SGD you would compute the average error for a batch and then backpropagate it. But then I was told in ...
8
votes
2answers
1k views

Can overfitting occur in Advanced Optimization algorithms?

while taking an online course on machine learning by Andrew Ng on coursera, I came across a topic called overfitting. I know it can occur when gradient descent is used in linear or logistic regression ...
8
votes
2answers
5k views

Understanding the mathematics of AdaGrad and AdaDelta

I have been building some models for a project, but I can't wrap my head around the math of Adagrad and Adadelta algorithms. I do understand how vanilla gradient descent works and I have written ...
7
votes
1answer
3k views

Why isn't leaky ReLU always preferable to ReLU given the zero gradient for x<0?

It looks to me like the leaky ReLU should have much better performance since the standard ReLU can’t use half of its space (x < 0 where the gradient is zero). But this doesn't happen and in ...
7
votes
2answers
18k views

Trying to understand Logistic Regression Implementation

I'm currently using the following code as a starting point to deepen my understanding of regularized logistic regression. As a first pass I'm just trying to do a binary classification on part of the ...
7
votes
1answer
12k views

What is the difference between SGD classifier and the Logisitc regression?

To my understanding, the SGD classifier, and Logistic regression seems similar. An SGD classifier with loss = 'log' implements Logistic regression and loss = 'hinge' implements Linear SVM. I also ...
7
votes
2answers
8k views

How to plot cost versus number of iterations in scikit learn?

One of the recommendations in the Coursera Machine Learning course when working with gradient descent based algorithms is: Debugging gradient descent. Make a plot with number of iterations on the x-...
7
votes
2answers
1k views

Why averaging the gradient works in Gradient Descent?

In Full-batch Gradient descent or Minibatch-GD we are getting gradient from several training examples. We then average them out to get a "high-quality" gradient, from several estimations and finally ...
7
votes
1answer
10k views

Implementation of Stochastic Gradient Descent in Python

I am attempting to implement a basic Stochastic Gradient Descent algorithm for a 2-d linear regression in python. I was given some boilerplate code for vanilla GD, and I have attempted to convert it ...
6
votes
2answers
3k views

Why do we use gradients instead of residuals in Gradient Boosting?

I have found mentions of two advantages in using gradients instead of actual residuals: 1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base ...
6
votes
2answers
120 views

Duplicated features for gradient descent

Suppose that our data matrix X has a duplicated column, i.e, there is a duplicated feature and the matrix is not full column rank. What happpens? I guess that we can not find a unique solution ...
6
votes
1answer
6k views

How does Gradient Descent and Backpropagation work together?

Please forgive me as I am new to this. I have attached a diagram trying to model my understanding of neural network and Back-propagation? From videos on Coursera and resources online I formed the ...
6
votes
1answer
577 views

Final layer of neural network responsible for overfitting

I am using a multi-layer perceptron with 2 hidden layers to solve a binary classification task on a noisy timeseries dataset with a class imbalance of 80/20. I have 30 million rows and 500 features in ...
5
votes
2answers
1k views

How to update weights in a neural network using gradient descent with mini-batches?

[I've cross-posted it to cross.validated because I'm not sure where it fits best] How does gradient descent work for training a neural network if I choose mini-batch (i.e., sample a subset of the ...
5
votes
1answer
1k views

What is conjugate gradient descent?

What is Conjugate Gradient Descent of Neural Network? How is it different from Gradient Descent technique? I came across a resource, but was unable to understand the difference between the two ...
5
votes
3answers
4k views

Stochastic gradient descent in logistic regression

I am very new to machine learning and in my first project have stumbled across a lot of issues which I really want to get through. I'm using logistic regression with R's ...
5
votes
1answer
525 views

Why is vanishing gradient a problem?

Let us say we are using a neural network with $4$ layers with $50,30,20,10$ neurons each. The problem of vanishing gradient would mean that the rate of change of parameters associated with the earlier ...
5
votes
2answers
4k views

How to get out of local minimums on stochastic gradient descent?

I'm not programming a neural network but I'm looking at it from a non-hands-on, theoretical point of view and I'm currently wondering how to escape a local minimum and how to get to a global minimum. ...
5
votes
1answer
3k views

Implementing RMSProp, but finding differences between reference versions

I am researching to implement RMSProp in a neural network project I am writing. I have not found any published paper to refer for a canonical version - I first stumbled across the idea from a ...
5
votes
1answer
3k views

clipping the reward for adam optimizer in keras

I would like to clip the reward in keras. I saw it is possible to clip the norm and clip the value is sgd as follows: ...
5
votes
2answers
151 views

Gradient Checking LSTM - how to get change in Cost across timesteps?

I am performing gradient check for my LSTM which has 4 timesteps. The LSTM looks as follows: ...
5
votes
1answer
291 views

How backpropagation through gradient descent represents the error after each forward pass

In Neural NEtwork Multilayer Perceptron, I understand that the main difference between Stochastic Gradient Descent (SGD) vs Gradient Descent (GD) lies in the way of how many samples are chosen while ...
5
votes
1answer
1k views

How does Tensorflow compute gradients of reduce_min operation?

With a non-differentiable operation, such as a minimization, how does Tensorflow compute the gradients? Some kind of soft-minimum approximation? If so, can I retrieve the analytical computation for a ...
5
votes
2answers
1k views

Gradient Descent in ReLU Neural Network

I’m new to machine learning and recently facing a problem on back propagation of training a neural network using ReLU activation function shown in the figure. My problem is to update the weights ...
5
votes
0answers
462 views

differences between LSQR and FTRL when working with very sparse data

I have a 2M instances dataset with millions of very very sparse dummy variables created using the hashing trick = ...
4
votes
1answer
3k views

What feature engineering is necessary with tree based algorithms?

I understand data hygiene, which is probably the most basic feature engineering. That is making sure all your data is properly loaded, making sure N/As are treated ...
4
votes
1answer
1k views

Why is stochastic gradient descent so much worse than batch GD for MNIST task?

Here the code from Tensorflow tutorial: A Multilayer Perceptron implementation example With batch size = 100 we quickly got Accuracy: 94.59%. If I set the batch size to be one, the training takes ...
4
votes
1answer
5k views

How does LightGBM deal with value scale?

I understand that the loss metric can be used as linear, or log, or other things. This is documented at http://lightgbm.readthedocs.io/en/latest/Parameters.html?highlight=logloss#metric-parameters I ...
4
votes
2answers
5k views

Plots with shaded standard deviation

What tools can I use to make a visualization similar to this one? I want to have the mean be bolded and the standard deviation be shaded.
4
votes
1answer
677 views

Why does gradient descent gives me much better Relative Squared Error then the Least Squares approach?

Am I doing regression task with 7 dependent variables and 10000 data points. The SGD gives me 22% of mean absolute percentage error on test and train dataset. And Least Squares method using numpy ...
4
votes
2answers
55 views

Question of using gradient descent instead of calculus. I checked previous questions there are still points to clarify

First of all I checked http://stats.stackexchange.com/questions/23128/solving-for-regression-parameters-in-closed-form-vs-gradient-descent, http://stackoverflow.com/questions/26804656/why-do-we-use-...
4
votes
1answer
5k views

Benefits of stochastic gradient descent besides speed/overhead and their optimization

Say I am training a neural network and can fit all my data into memory. Are there any benefits to using mini batches with SGD in this case? Or is batch training with the full gradient always superior ...
4
votes
1answer
510 views

Does Gradient Boosting detect non-linear relationships?

I wish to train some data using the the Gradient Boosting Regressor of Scikit-Learn. My questions are: 1) Is the algorithm able to capture non-linear relationships? For example, in the case of y=x^2,...
4
votes
2answers
2k views

Using Mean Squared Error in Gradient Descent

I've recently been writing linear regression algorithms from scratch to gain an understanding of how the maths behind it works (something that was a bit of a black box beforehand), and so I got around ...
4
votes
1answer
415 views

How to search for an optimal dithering pattern?

I'm trying to find an optimal dithering pattern which can be used as a threshold on a greyscale image to generate a 1 bit black and white image. Ideally it would be optimal in the sense that a human ...
4
votes
1answer
264 views

Why do CNNs with ReLU learn that well?

Convolutional Neural Networks (CNNs) use almost always the rectified linear activation function (ReLU): $$f(x) = max(0, x)$$ However, the derivative of this function is $$f'(x) = \begin{cases} 0 &...
4
votes
2answers
2k views

When is Gradient Descent invoked on the objective function while running XGboost?

is it at the end of every tree? or only after all trees are build? I tried to think in both ways but didn't get a clear picture. Can we focus more the part "the loss function is applied between ...
4
votes
3answers
1k views

Training Restricted Boltzmann Machines (RBMs) using gradient descent

Hey I am a little new to the whole RBM entropy/energy training thing, having some trouble understanding and trying to figure whether it is worth the effort needed to understand. Can't RBMs quite ...
4
votes
2answers
1k views

Is empirical risk the same thing as loss function?

I am reading the article Stochastic Gradient Descent Tricks by Léon Bottou (avaible here) and on the very first page they introduce empirical risk $E_n(f) = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),y_i),$ ...
4
votes
2answers
110 views

Why Gradient methods work in finding the parameters in Neural Networks?

After reading quite a lot of papers (20-30 or so), I feel that I am quite not understanding things. Let us focus on the supervised learnings (for example). Given a set of data $\mathcal{D}_{train}=\{...

1
2 3 4 5
7