# Neural Network Learning Rate vs Q-Learning Learning Rate

I'm just getting into machine learning--mostly Reinforcement Learning--using a neural network trained on Q-values. However, in looking at the hyper-parameters, there are two that seem redundant: the learning rate for the neural network, $\eta$, and the learning rate for Q-learning, $\alpha$. They both seem to change the rate at which the neural net takes new conclusions over old ones.

So are these two parameters redundant? Do I need to worry about even having $\alpha$ as anything other than 1 if I'm already tuning $\eta$, or do they have ultimately different effects?

• Are you reading your descriptions of neural networks and reinforcement learning from two different sources, and trying to combine the theory yourself, or reading an article that is specifically about combining the two? If the latter, could you give a reference, in case the article has reserved any special meaning for the two different learning rates? Commented Aug 13, 2017 at 16:02
• A couple places talk about combining them but most never talk about the learning rate on Q-Learning, which is brought up on the Wikipedia page as a good idea for stochastic learning environments (which I plan to use). So using both is my idea.
– Jake
Commented Aug 15, 2017 at 18:22

There is usually only one learning rate active when using a neural network as function approximator in reinforcement learning. The different names $\eta$ and $\alpha$ are just different conventions for the same basic concept.

When you use a function approximator, other than a linear one, in reinforcement learning, then typically you would not use TD error based update like this:

$$\mathbf{w} \leftarrow \mathbf{w} + \alpha[R+\gamma\hat{q}(S', A',\mathbf{w}) - \hat{q}(S, A,\mathbf{w})]\nabla \hat{q}(S, A,\mathbf{w})$$

But you would train your estimator in a supervised learning manner on sampled TD target (which would use $\eta$ param in a neural network):

$$\mathbf{x} = \phi(S, A), y = R+\gamma\hat{q}(S', A',\mathbf{w})$$

You can actually do either if your library supports it - it is certainly possible to calculate $\nabla \hat{q}(S, A,\mathbf{w})$ for a neural network for instance, instead of using an explicit training loss function. However, the two approaches are equivalent ways of expressing the same thing, there is no reason to use both.

There are other parameters used in reinforcement learning that may affect rates of convergence and other properties of learning agents. For example with differential semi-gradient TD learning - which might be an algorithm you would look at for a continuous task - Sutton and Barto present $\beta$ as a separate learning rate for the average reward, distinct from the learning rate of the estimator.

So are these two parameters redundant? Do I need to worry about even having $\alpha$ as anything other than 1 if I'm already tuning $eta$, or do they have ultimately different effects?

They are essentially the same parameter with a different name.

If you are trying to pass an error value like $R+\gamma\hat{q}(S', A',\mathbf{w}) - \hat{q}(S, A,\mathbf{w})$ (whether multiplied by $\alpha$ or not) into the neural network as a target, then you have got the wrong idea. Instead you want your neural network to learn the target $R+\gamma\hat{q}(S', A',\mathbf{w})$. That's because the subtraction of current prediction and multiplication by a learning rate is built into the neural network training.