Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [learning-rate]

The tag has no usage guidance.

0
votes
0answers
8 views

How to find learning rate decay?

Given the number of epochs, batch size and learning rate, is there a formula by which I can calculate the learning rate decay in mini batch SGD?
2
votes
2answers
43 views

Constant Learning Rate for Gradient Decent

Given, we have a learning rate, $\alpha_n$ for the $n^{th}$ step of the gradient descent process. What would be the impact of using a constant value for $\alpha_n$ in gradient descent?
1
vote
1answer
41 views

Is it necessary to tune the step size, when using Adam?

The Adam optimizer has four main hyperparameters. For example, looking at the Keras interface, we have: ...
2
votes
1answer
175 views

Constant validation loss & accuracy, training accuracy fluctuates

I am training a Squeeze-net model for binary classification of images. I have 79968 images for training (50:50 for and against) and 8892 images in the validation set. After 35000 iterations my ...
2
votes
1answer
335 views

Is it a good practice to always apply `ReduceLROnPlateau()`, given that models benefit from reducing learning rate once learning stagnates?

The rationale behind the keras function ReduceLROnPlateau() is that models benefit from reducing learning rate once learning stagnates. Is it a good practice to ...
2
votes
1answer
38 views

Why are optimization algorithms slower at critical points?

I just found the animation below from Alec Radford's presentation: As visible, all algorithms are considerably slowed down at saddle point (where derivative is 0) and quicken up once they get out of ...
0
votes
0answers
125 views

Fast.ai learning rate proposal

In the fast.ai deep learning course, Jeremy Howard mentions a paper on how to choose the learning rate for training a neural network. This learning rate criterium is explained in this blog post, as ...
0
votes
1answer
77 views

Learning Rate based on error of the network

I am not an expert and do not have theoretical justification for that, but it seems to me that the smaller network error is, the smaller learning rate should be. Is there an algorithm to dynamically ...
1
vote
0answers
54 views

The Gradient descent different between in Ng coursera and Michael A. Nielsen book

I am learning the neural networking from NG machine learning course in coursera and the book neural networking and deep learning by Nielson. I have a little confusion about the understanding of the ...
1
vote
0answers
84 views

How to decrease the learning rate only when cost function is stagnant?

I'm training a very complex function in tensorflow. Is there a way to decrease the learning rate only when the cost isin't decreasing very much?
4
votes
1answer
542 views

When should you use learning rate scheduling over adaptive learning rate optimization algorithm?

In order to converge to the optimum properly, there have been invented different algorithms that use adaptive learning rate, such as AdaGrad, Adam, and RMSProp. On the other hand, there is a learning ...