We’re rewarding the question askers & reputations are being recalculated! Read more.

# Questions tagged [learning-rate]

The tag has no usage guidance.

16 questions
Filter by
Sorted by
Tagged with
26 views

### Learning rate Scheduler

A very important aspect in deep learning is the learning rate. Can someone tell me, how to initialize the lr and how to choose ...
5 views

### Does accuracy scale logarthmic with Learningrates?

This is really general, but: How influential is a relatively small change in learning rate for any given algorithm? influences a change from 0.1 to 0.11 in similar magnitude as 0.0001 to 0.1001? I ...
16 views

### Slowly decreasing validation / training cost and their abnormal values

I have a dataset of size ~100,000 of images, I'm training a CNN model on them for regression. optimizer: Adam batch_size: 64 Number of epochs: 50 When I set the ...
20 views

### Is a small but constant learning rate better for generalization than learning rate decay?

Learning rate decay - starting with a higher learning rate for fast convergence and then decreasing the learning rate for better convergence - allows training loss to converge to the same value in ...
59 views

### GAN optimizer settings in Keras

I am working on a Generative Adversarial Network, implementing in Keras. I have my generator model, G, and discriminator D, both are being created by two functions, and then the GAN model is created ...
2k views

### How to see/change learning rate in Keras LSTM?

I see in some question/answers that they say decrease the learning rate. But I don't know how can I see and change the learning rate of LSTM model in ...
39 views

### How to find learning rate decay?

Given the number of epochs, batch size and learning rate, is there a formula by which I can calculate the learning rate decay in mini batch SGD?
108 views

### Constant Learning Rate for Gradient Decent

Given, we have a learning rate, $\alpha_n$ for the $n^{th}$ step of the gradient descent process. What would be the impact of using a constant value for $\alpha_n$ in gradient descent?
162 views

### Is it necessary to tune the step size, when using Adam?

The Adam optimizer has four main hyperparameters. For example, looking at the Keras interface, we have: ...
331 views

### Constant validation loss & accuracy, training accuracy fluctuates

I am training a Squeeze-net model for binary classification of images. I have 79968 images for training (50:50 for and against) and 8892 images in the validation set. After 35000 iterations my ...
972 views

### Is it a good practice to always apply ReduceLROnPlateau(), given that models benefit from reducing learning rate once learning stagnates?

The rationale behind the keras function ReduceLROnPlateau() is that models benefit from reducing learning rate once learning stagnates. Is it a good practice to ...
63 views

### Why are optimization algorithms slower at critical points?

I just found the animation below from Alec Radford's presentation: As visible, all algorithms are considerably slowed down at saddle point (where derivative is 0) and quicken up once they get out of ...
93 views

### Learning Rate based on error of the network

I am not an expert and do not have theoretical justification for that, but it seems to me that the smaller network error is, the smaller learning rate should be. Is there an algorithm to dynamically ...
55 views

### The Gradient descent different between in Ng coursera and Michael A. Nielsen book

I am learning the neural networking from NG machine learning course in coursera and the book neural networking and deep learning by Nielson. I have a little confusion about the understanding of the ...