We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [learning-rate]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
26 views

Learning rate Scheduler

A very important aspect in deep learning is the learning rate. Can someone tell me, how to initialize the lr and how to choose ...
0
votes
0answers
5 views

Does accuracy scale logarthmic with Learningrates?

This is really general, but: How influential is a relatively small change in learning rate for any given algorithm? influences a change from 0.1 to 0.11 in similar magnitude as 0.0001 to 0.1001? I ...
1
vote
0answers
16 views

Slowly decreasing validation / training cost and their abnormal values

I have a dataset of size ~100,000 of images, I'm training a CNN model on them for regression. optimizer: Adam batch_size: 64 Number of epochs: 50 When I set the ...
0
votes
0answers
20 views

Is a small but constant learning rate better for generalization than learning rate decay?

Learning rate decay - starting with a higher learning rate for fast convergence and then decreasing the learning rate for better convergence - allows training loss to converge to the same value in ...
1
vote
0answers
59 views

GAN optimizer settings in Keras

I am working on a Generative Adversarial Network, implementing in Keras. I have my generator model, G, and discriminator D, both are being created by two functions, and then the GAN model is created ...
0
votes
1answer
2k views

How to see/change learning rate in Keras LSTM?

I see in some question/answers that they say decrease the learning rate. But I don't know how can I see and change the learning rate of LSTM model in ...
0
votes
1answer
39 views

How to find learning rate decay?

Given the number of epochs, batch size and learning rate, is there a formula by which I can calculate the learning rate decay in mini batch SGD?
2
votes
2answers
108 views

Constant Learning Rate for Gradient Decent

Given, we have a learning rate, $\alpha_n$ for the $n^{th}$ step of the gradient descent process. What would be the impact of using a constant value for $\alpha_n$ in gradient descent?
1
vote
1answer
162 views

Is it necessary to tune the step size, when using Adam?

The Adam optimizer has four main hyperparameters. For example, looking at the Keras interface, we have: ...
2
votes
1answer
331 views

Constant validation loss & accuracy, training accuracy fluctuates

I am training a Squeeze-net model for binary classification of images. I have 79968 images for training (50:50 for and against) and 8892 images in the validation set. After 35000 iterations my ...
2
votes
1answer
972 views

Is it a good practice to always apply `ReduceLROnPlateau()`, given that models benefit from reducing learning rate once learning stagnates?

The rationale behind the keras function ReduceLROnPlateau() is that models benefit from reducing learning rate once learning stagnates. Is it a good practice to ...
2
votes
1answer
63 views

Why are optimization algorithms slower at critical points?

I just found the animation below from Alec Radford's presentation: As visible, all algorithms are considerably slowed down at saddle point (where derivative is 0) and quicken up once they get out of ...
0
votes
1answer
93 views

Learning Rate based on error of the network

I am not an expert and do not have theoretical justification for that, but it seems to me that the smaller network error is, the smaller learning rate should be. Is there an algorithm to dynamically ...
1
vote
0answers
55 views

The Gradient descent different between in Ng coursera and Michael A. Nielsen book

I am learning the neural networking from NG machine learning course in coursera and the book neural networking and deep learning by Nielson. I have a little confusion about the understanding of the ...
1
vote
0answers
99 views

How to decrease the learning rate only when cost function is stagnant?

I'm training a very complex function in tensorflow. Is there a way to decrease the learning rate only when the cost isin't decreasing very much?
4
votes
1answer
839 views

When should you use learning rate scheduling over adaptive learning rate optimization algorithm?

In order to converge to the optimum properly, there have been invented different algorithms that use adaptive learning rate, such as AdaGrad, Adam, and RMSProp. On the other hand, there is a learning ...