3
$\begingroup$

Why does action exploration in DQN not lead to instability?

I see in DQN algorithms, that it selects random actions even after some iterations.

My question is how does this approach not lead to instability? Even the final value of epsilon (the probability of a random action) is non-zero!

If we set the final epsilon=0.01, we will select a random the action one time in a hundred, which means that we are not going to get the same answer in different runs and probably our answers are not stable. Why does this still work?

$\endgroup$
3
$\begingroup$

All Reinforcement Learning (RL) control algorithms have to deal with the exploration/exploitation trade-off. Do you trust that you have learned enough about the environment that you can act optimally, or do you still need to check alternative actions.

Related to this, there are two basic approaches when trying to learn an optimal policy in RL. An agent can either learn:

  • on-policy where the agent assesses the actions that it is taking, and adjusts behaviour directly. To solve the exploration/exploration trade-off with on-policy learning, a common approach is to decay $\epsilon$ (or a similar parameter which controls the amount of exploration), based on some idea of how much learning is necessary to learn the task.

  • off-policy where the agent has separate behaviour and target policies. The agent takes actions from an exploring behaviour policy, and adjusts the results it gets mathematically so that it calculates a different target policy, which is often optimal in some way.

Q learning is an off-policy algorithm, which learns the action value function of following deterministic greedy action choices at all times. In the simplest single-step version, the adjustment required is really simple - instead of estimating the action value based on what the behaviour policy does, Q learning updates using whatever the best next action would be, regardless of what gets chosen.

Q learning is proven to be stable and to converge when using a table of q values (i.e. not a neural network). However it is actually not very stable when used with neural networks for estimating $\hat{q}(s,a, \theta)$ (where $\theta$ is the NN weights). This is why DQN needs to use experience replay and a separate target network - in order to combat the instability. The instability is not directly caused by random actions though, it is more complex than that.

If we set the final epsilon=0.01, we will select a random the action one time in a hundred, which means that we are not going to get the same answer in different runs and probably our answers are not stable. Why does this still work?

The agent will not behave consistently on each run, as it is still exploring. However, if it has been set up correctly, it will still converge to and learn an optimal policy, and in fact the constant exploration should help it get values for non-optimal action choices correct, and maintain the correct maximising actions.

When you use a Q learning policy after it has finished learning, typically you would set epsilon to zero and have the agent act greedily.

It is worth noting that even with a deterministic policy, RL is designed around the concept of an environment where random events can happen. For instance it will learn the best way to move when navigating, even if sometimes the distance travelled is not the same due to random events. When you test an agent in a non-deterministic environment, then different things will happen each time, and this is not a problem for the learning process. It might be a problem for you to evaluate the agent - you would need to run a test multiple times and take an average to assess how well it is performing.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks Neil, this means that final no exploration after finding good setting? $\endgroup$ – user10296606 Sep 11 '18 at 0:08
  • $\begingroup$ @user10296606: Yes you can stop exploring in Q learning when you want to use the action values it has learned in a greedy policy. When you do that you also typically stop the learning algorithm, just query Q(s,a) and find maximising value of a e.g. $\pi(s) = \text{argmax}_a \hat{q}(s,a,\theta)$ $\endgroup$ – Neil Slater Sep 11 '18 at 6:31
  • $\begingroup$ thanks, if for some episodes in training it leads to a good results, but for a larger amount of episodes its results is not a good answer, what we can say and what we can do to solve it? $\endgroup$ – user10296606 Sep 11 '18 at 16:14
  • $\begingroup$ @user10296606: Sorry I don't understand your comment. It seems like a different question though, you don't appear to be asking about epsilon and stability any more? If that's the case, and this answer helped you, then please accept the answer (click on the tick mark) and ask a new question $\endgroup$ – Neil Slater Sep 11 '18 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.