Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
46 votes

Does gradient descent always converge to an optimum?

Gradient Descent is an algorithm which is designed to find the optimal points, but these optimal points are not necessarily global. And yes if it happens that it diverges from a local location it may ...
Green Falcon's user avatar
  • 14.1k
42 votes

Why not always use the ADAM optimization technique?

Here’s a blog post reviewing an article claiming SGD is a better generalized adapter than ADAM. There is often a value to using more than one method (an ensemble), because every method has a weakness.
Christopher Klaus's user avatar
40 votes

Should a model be re-trained if new observations are available?

When new observations are available, there are three ways to retrain your model: Online: each time a new observation is available, you use this single data point to further train your model (e.g. ...
tombarti's user avatar
  • 501
33 votes

Should a model be re-trained if new observations are available?

Once a model is trained and you get new data which can be used for training, you can load the previous model and train onto it. For example, you can save your model as a ...
Hima Varsha's user avatar
  • 2,346
31 votes

Are there any rules for choosing the size of a mini-batch?

In On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima there are a couple of intersting statements: It has been observed in practice that when using a larger batch ...
30 votes

Is Gradient Descent central to every optimizer?

No. Gradient descent is used in optimization algorithms that use the gradient as the basis of its step movement. Adam, Adagrad, ...
jeb02's user avatar
  • 416
27 votes

Difference between RMSProp with momentum and Adam Optimizers

(My answer is based mostly on Adam: A Method for Stochastic Optimization (the original Adam paper) and on the implementation of rmsprop with momentum in Tensorflow (which is operator() of struct ...
Oren Milman's user avatar
23 votes

How many features to sample using Random Forests

I think in the original paper they suggest using $\log_2(N +1$), but either way the idea is the following: The number of randomly selected features can influence the generalization error in two ways: ...
oW_'s user avatar
  • 6,357
21 votes

Does gradient descent always converge to an optimum?

Asides from the points you mentioned (convergence to non-global minimums, and large step sizes possibly leading to non-convergent algorithms), "inflection ranges" might be a problem too. Consider the ...
Ami Tavory's user avatar
  • 1,277
20 votes

local minima vs saddle points in deep learning

Let me give an explanation based on multivariate calculus. If you have taken a multivariate course, you will have heard that, given a critical point (point where the gradient is zero), the condition ...
David Masip's user avatar
  • 6,081
18 votes

Guidelines for selecting an optimizer for training neural networks

AdaGrad penalizes the learning rate too harshly for parameters which are frequently updated and gives more learning rate to sparse parameters, parameters that are not updated as frequently. In several ...
Santanu_Pattanayak's user avatar
14 votes

local minima vs saddle points in deep learning

This is simply trying to convey my intuition, i.e. no rigor. The thing with saddle points is that they are a type of optimum which combines a combination of minima and maxima. Because the number of ...
user41985's user avatar
  • 156
11 votes

Can overfitting occur in Advanced Optimization algorithms?

There is no technique that will eliminate the risk of overfitting entirely. The methods you've listed are all just different ways of fitting a linear model. A linear model will have a global minimum, ...
Ryan Zotti's user avatar
  • 4,149
10 votes

Choosing a learning rate

Copy-pasted from my masters thesis: If the loss does not decrease for several epochs, the learning rate might be too low. The optimization process might also be stuck in a local minimum. Loss being ...
Martin Thoma's user avatar
  • 18.9k
9 votes

Why is learning rate causing my neural network's weights to skyrocket?

You might find Chapter 8 of Deep Learning helpful. In it, the authors discuss training of neural network models. It's very intricate, so I'm not surprised you're having difficulties. One possibility (...
vbox's user avatar
  • 341
9 votes

Why is my generator loss function increasing with iterations?

I think that there are several issues with your model: First of all - Your generator's loss is not the generator's loss. You have on binary cross-entropy loss function for the discriminator, and you ...
Mark.F's user avatar
  • 2,220
8 votes

Why do we use gradients instead of residuals in Gradient Boosting?

Hmmm, I am little perplexed by your question. In gradient boosting, we do use the residuals. The residuals are the gradients. You can check my simple implementation of gradient boosting. This is ...
Ricardo Cruz's user avatar
  • 3,420
8 votes

How does binary cross entropy work?

When doing logistic regression you start calculating a bunch of probabilities $p_i$ and your target is maximize the product of those probabilities (as they're considered independent events). The ...
Alberto's user avatar
  • 251
8 votes

Why aren't Genetic Algorithms used for optimizing neural networks?

Training Neural Networks (NNs) with Genetic Algorithms (GAs) is not only feasible, there are some niche areas where the performance is good enough to be used frequently. A good example of this is ...
Neil Slater's user avatar
8 votes

Is Gradient Descent central to every optimizer?

According to the title: No. Only specific types of optimizers are based on Gradient Descent. A straightforward counterexample is when optimization is over a discrete space where gradient is undefined. ...
Esmailian's user avatar
  • 9,322
8 votes

How many times is backprop used in epoch?

It depends on the type of gradient descent or respectively your batch size: One epoch means that your neural net (NN) has applied the forward pass on all examples of your training data, i.e. it has "...
Jonathan's user avatar
  • 5,410
8 votes

Difference between RMSProp and Momentum?

Optimizers evolved with small Fix/Improvement on the previous one. So, if you will read in sequence, you will have a better understanding. In this context, RMSProp was a fix on Adagrad and it was an ...
10xAI's user avatar
  • 5,624
7 votes

Does gradient descent always converge to an optimum?

Conjugate gradient is not guaranteed to reach a global optimum or a local optimum! There are points where the gradient is very small, that are not optima (inflection points, saddle points). Gradient ...
Herbert Knieriem's user avatar
7 votes

Is reseating passengers a reinforcement learning problem?

Reinforcement learning is more about interacting with an environment, and while this could be posed as an RL problem, I think using Global Optimization would be a more direct approach. Essentially ...
Imran's user avatar
  • 2,381
7 votes

Mathematical formulation of Support Vector Machines?

Your understandings are right. deriving the margin to be $\frac{2}{|w|}$ we know that $w \cdot x +b = 1$ If we move from point z in $w \cdot x +b = 1$ to the $w \cdot x +b = 0$ we land in a ...
Fatemeh Asgarinejad's user avatar
7 votes

Is it possible to make F1_Score differentiable and use it directly as a Loss function?

Yes there is, let's take $F_1$ score base definition, with : $$ F_1 = 2 \times \frac{precision \times recall} {precision + recall} \\ F_1 = \frac{2 \times TP} {2 \times TP + FP + FN} $$ And this is ...
Thomas FEL's user avatar
7 votes

Is it possible to get worse model after optimization?

Is it possible that after running the optimization my score won't get better (and even worse?) ? Yes, theoretically, by pure luck, it is possible that your initial guess, before optimization of hyper-...
aivanov's user avatar
  • 1,520
6 votes

Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training?

Yes the ReLU second order derivative is 0. Technically, neither $\frac{dy}{dx}$ nor $\frac{d^2y}{dx^2}$ are defined at $x=0$, but we ignore that - in practice an exact $x=0$ is rare and not especially ...
Neil Slater's user avatar
6 votes

Should a model be re-trained if new observations are available?

When should you re-train? Theoretically, a model will only degrade (become outdated and no longer useful) if the system you are modelling or the nature of the data has changed. Ideally you can spot ...
Renel Chesak's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible