86 votes

What is the difference between Gradient Descent and Stochastic Gradient Descent?

For a quick simple explanation: In both gradient descent (GD) and stochastic gradient descent (SGD), you update a set of parameters in an iterative manner to minimize an error function. While in GD, ...
Sociopath's user avatar
  • 1,233
45 votes
Accepted

Does gradient descent always converge to an optimum?

Gradient Descent is an algorithm which is designed to find the optimal points, but these optimal points are not necessarily global. And yes if it happens that it diverges from a local location it may ...
Green Falcon's user avatar
31 votes
Accepted

Why ReLU is better than the other activation functions

The biggest advantage of ReLu is indeed non-saturation of its gradient, which greatly accelerates the convergence of stochastic gradient descent compared to the sigmoid / tanh functions (paper by ...
Maxim's user avatar
  • 890
30 votes
Accepted

Is Gradient Descent central to every optimizer?

No. Gradient descent is used in optimization algorithms that use the gradient as the basis of its step movement. Adam, Adagrad, ...
jeb02's user avatar
  • 416
23 votes
Accepted

Scikit-learn: Getting SGDClassifier to predict as well as a Logistic Regression

The comments about iteration number are spot on. The default SGDClassifier n_iter is 5 ...
cwharland's user avatar
  • 961
20 votes
Accepted

How does Gradient Descent and Backpropagation work together?

First, remember that the derivative of a function gives the direction in which the function increases, and its negative, the direction in which the function decreases. Training a model is just ...
Escachator's user avatar
19 votes

Does gradient descent always converge to an optimum?

Asides from the points you mentioned (convergence to non-global minimums, and large step sizes possibly leading to non-convergent algorithms), "inflection ranges" might be a problem too. Consider the ...
Ami Tavory's user avatar
  • 1,237
19 votes

What is the difference between Gradient Descent and Stochastic Gradient Descent?

The inclusion of the word stochastic simply means the random samples from the training data are chosen in each run to update parameter during optimisation, within the framework of gradient descent. ...
n1k31t4's user avatar
  • 14.8k
17 votes
Accepted

What is momentum in neural network?

Momentum in neural networks is a variant of the stochastic gradient descent. It replaces the gradient with a momentum which is an aggregate of gradients as very well explained here. It is also the ...
etiennedm's user avatar
  • 1,385
15 votes
Accepted

What is the difference between SGD classifier and the Logisitc regression?

Welcome to SE:Data Science. SGD is a optimization method, while Logistic Regression (LR) is a machine learning algorithm/model. You can think of that a machine learning model defines a loss function, ...
user12075's user avatar
  • 2,254
13 votes
Accepted

Why isn't leaky ReLU always preferable to ReLU given the zero gradient for x<0?

One reason that ReL Units have been introduced is to circumvent the problem of vanishing gradients of sigmoidal units at -1 and 1. Another advantage of ReL Units is that they saturate at exactly 0 ...
oW_'s user avatar
  • 6,327
13 votes
Accepted

Why averaging the gradient works in Gradient Descent?

Each training sample ends up in a distant, completely separate location on the error-surface That is not a correct visualisation of what is going on. The error surface plot is tied to the value of ...
Neil Slater's user avatar
  • 28.8k
12 votes
Accepted

What feature engineering is necessary with tree based algorithms?

Feature engineering that I would consider essential for even tree based algorithms are: Modular arithmetic calculations: e.g. converting a timestamp into day of the week, or time of day. If your ...
Eumenedies's user avatar
12 votes
Accepted

Using a random forest, would a RandomForest performance be less if I drop the first or the last tree?

The two slightly-smaller models will perform exactly the same, on average. There is no difference baked in to the different trees: "the last tree will be the best trained" is not true. The ...
Ben Reiniger's user avatar
  • 11.6k
12 votes
Accepted

How to fit a math formula to data?

If you know $k$, which it seems you do, then this is just a linear regression. In fact, with just one feature (the $x^k$), this is a simple linear regression, and easy equations apply without you ...
Dave's user avatar
  • 3,688
11 votes
Accepted

Can overfitting occur in Advanced Optimization algorithms?

There is no technique that will eliminate the risk of overfitting entirely. The methods you've listed are all just different ways of fitting a linear model. A linear model will have a global minimum, ...
Ryan Zotti's user avatar
  • 4,139
10 votes

What is the difference between Gradient Descent and Stochastic Gradient Descent?

In Gradient Descent or Batch Gradient Descent, we use the whole training data per epoch whereas, in Stochastic Gradient Descent, we use only single training example per epoch and Mini-batch Gradient ...
Rajat Gupta's user avatar
9 votes
Accepted

Why is learning rate causing my neural network's weights to skyrocket?

You might find Chapter 8 of Deep Learning helpful. In it, the authors discuss training of neural network models. It's very intricate, so I'm not surprised you're having difficulties. One possibility (...
vbox's user avatar
  • 341
9 votes
Accepted

How does LightGBM deal with value scale?

Generally, in tree-based models the scale of the features does not matter. This is because at each tree level, the score of a possible split will be equal whether the respective feature has been ...
Stergios's user avatar
  • 300
9 votes

How to prevent vanishing gradient or exploding gradient?

Vanishing gradient and exploding gradient are two common effects associated to training deep neural networks and their impact is usually stronger the deeper the network. As you know, two fundamental ...
juanba1984's user avatar
8 votes

Why do we use gradients instead of residuals in Gradient Boosting?

Hmmm, I am little perplexed by your question. In gradient boosting, we do use the residuals. The residuals are the gradients. You can check my simple implementation of gradient boosting. This is ...
Ricardo Cruz's user avatar
  • 3,410
8 votes

Is Gradient Descent central to every optimizer?

According to the title: No. Only specific types of optimizers are based on Gradient Descent. A straightforward counterexample is when optimization is over a discrete space where gradient is undefined. ...
Esmailian's user avatar
  • 9,252
8 votes
Accepted

How many times is backprop used in epoch?

It depends on the type of gradient descent or respectively your batch size: One epoch means that your neural net (NN) has applied the forward pass on all examples of your training data, i.e. it has "...
Jonathan's user avatar
  • 5,400
8 votes

Difference between RMSProp and Momentum?

Optimizers evolved with small Fix/Improvement on the previous one. So, if you will read in sequence, you will have a better understanding. In this context, RMSProp was a fix on Adagrad and it was an ...
10xAI's user avatar
  • 5,574
7 votes
Accepted

How to update weights in a neural network using gradient descent with mini-batches?

Let us say that the output of one neural network given it's parameters is $$f(x;w)$$ Let us define the loss function as the squared L2 loss (in this case). $$L(X,y;w) = \frac{1}{2n}\sum_{i=0}^{n}[f(...
Armen Aghajanyan's user avatar
7 votes
Accepted

Why is stochastic gradient descent so much worse than batch GD for MNIST task?

Why is stochastic gradient descent so much worse then batch GD for MNIST task? It isn't inherently worse. Instead, by changing just one parameter on its own you have adjusted the example outside of ...
Neil Slater's user avatar
  • 28.8k
7 votes

Gradient descent and partial derivatives

What's the point? First, it is good to understand what we are doing that leads us to need these tools. When we are trying to apply machine learning we want to infer some meaning from data. This means ...
JahKnows's user avatar
  • 8,846
7 votes

What is the difference between Gradient Descent and Stochastic Gradient Descent?

Gradient Descent is an algorithm to minimize the $J(\Theta)$! Idea: For current value of theta, calculate the $J(\Theta)$, then take small step in direction of negative gradient. Repeat. Update ...
DRV's user avatar
  • 171
7 votes
Accepted

Duplicated features for gradient descent

In 'Efficient Backprop' by Lecun and others (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), they explain why correlated variables are bad (§ 4.3 normalizing the inputs). Duplicated data is a ...
Lucas Morin's user avatar
  • 2,093
7 votes
Accepted

Changing the batch size during training

Efficient use of resources It is a balancing game with the learning rate, and one reason you don't normally see people do this is that you want to utilise as much of the GPU as possible. It is ...
n1k31t4's user avatar
  • 14.8k

Only top scored, non community-wiki answers of a minimum length are eligible