Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [learning-rate]

The tag has no usage guidance.

1
vote
1answer
30 views

Is it necessary to tune the step size, when using Adam?

The Adam optimizer has four main hyperparameters. For example, looking at the Keras interface, we have: ...
0
votes
0answers
35 views

Help in tuning hyperparameters of DQN based on resulted stats

I'm not an expert in machine learning. I'm trying to make an agent learn to play in a Gym environment that I created. I'm not asking for help in debugging the code. I'm asking for help in interpreting ...
2
votes
1answer
97 views

Constant validation loss & accuracy, training accuracy fluctuates

I am training a Squeeze-net model for binary classification of images. I have 79968 images for training (50:50 for and against) and 8892 images in the validation set. After 35000 iterations my ...
1
vote
1answer
184 views

Is it a good practice to always apply `ReduceLROnPlateau()`, given that models benefit from reducing learning rate once learning stagnates?

The rationale behind the keras function ReduceLROnPlateau() is that models benefit from reducing learning rate once learning stagnates. Is it a good practice to ...
2
votes
1answer
31 views

Why are optimization algorithms slower at critical points?

I just found the animation below from Alec Radford's presentation: As visible, all algorithms are considerably slowed down at saddle point (where derivative is 0) and quicken up once they get out of ...
0
votes
0answers
100 views

Fast.ai learning rate proposal

In the fast.ai deep learning course, Jeremy Howard mentions a paper on how to choose the learning rate for training a neural network. This learning rate criterium is explained in this blog post, as ...
0
votes
1answer
72 views

Learning Rate based on error of the network

I am not an expert and do not have theoretical justification for that, but it seems to me that the smaller network error is, the smaller learning rate should be. Is there an algorithm to dynamically ...
1
vote
0answers
53 views

The Gradient descent different between in Ng coursera and Michael A. Nielsen book

I am learning the neural networking from NG machine learning course in coursera and the book neural networking and deep learning by Nielson. I have a little confusion about the understanding of the ...
1
vote
0answers
75 views

How to decrease the learning rate only when cost function is stagnant?

I'm training a very complex function in tensorflow. Is there a way to decrease the learning rate only when the cost isin't decreasing very much?
4
votes
1answer
478 views

When should you use learning rate scheduling over adaptive learning rate optimization algorithm?

In order to converge to the optimum properly, there have been invented different algorithms that use adaptive learning rate, such as AdaGrad, Adam, and RMSProp. On the other hand, there is a learning ...