# What is difference between final episodes of training and test in DQN?

What is difference between running in final episode of training mode and running in test mode in DQN?

Is there any difference more than after training and tune the hyper-parameters, we test for one episode and without any exploration? This means that test mode is similar to training mode in episode n+1 without exploring (while we train for n episode) ?Is it correct?

Why in some test code of DQN, they test for multiple episodes?

• I revised my explanation. episodes, I mean! – user10296606 Sep 29 '18 at 16:15

What is difference between running in final episode of training mode and running in test mode in DQN?

As Q learning is an off-policy method, and DQN is based on Q learning, then the agent effectively has two different policies:

• A behaviour policy that explores actions regardless of whether data so far suggests that they are optimal. The simplest and perhaps most commonly implemented behaviour policy is $$\epsilon$$-greedy with respect to action values $$\hat{q}(s,a,\theta)$$.

• A target policy that is being learned, which is the best guess at an optimal policy so far. In Q learning that policy is fully greedy with respect to action values $$\hat{q}(s,a,\theta)$$.

When assessing how well the agent has learned the environment after training, you are interested in the target policy. So with Q learning you need to run the environment with the agent choosing actions greedily. That could be achieved by setting $$\epsilon = 0$$, or with a different action selection routine which doesn't even check $$\epsilon$$ (which to use is an implementation detail).

Is there any difference more than after training and tune the hyper-parameters, we test for one episode and without any exploration?

Essentially yes to second part - without exploration. But . . .

Why in some test code of DQN, they test for multiple episodes?

That is because one episode is often not enough to get reliable statistics.

If the environment is completely deterministic, including having only one start state, then one episode would be sufficient to assess an agent.

However, if the environment is stochastic, or you are interested in multiple possible start states, then multiple episodes are necessary in order to have low mean squared error bounds on the average total return, or any other metrics you might be interested in. Otherwise it would be hard to compare results between different hyperparameters.

How many episodes are enough? There is no fixed number, it depends on how much variance there is in the environment and how accurately you want to know the values. In some cases - e.g. OpenAI gym, there are suggested values. For instance, LunarLander suggests a target score over 100 episodes, so they are suggesting measurements are accurate enough over 100 episodes. However, the advice only applies to that problem. In general, you can measure the variance in the score, like any other statistical measure, and calculate error bounds on your metric.

• Thank Neil and commonly how many episodes is enough in test case? I have some problem about difference between episode and epoch! Episode=1 or k(k= multiple epoch) that after this k number of BP we have one episode? – user10296606 Sep 29 '18 at 17:33
• @user10296606: There is no fixed number, it depends on how much variance there is in the environment and how accurately you want to know the values. In some cases - e.g. OpenAI gym, there are suggested values - e.g. LunarLander suggests a target score over 100 episodes, so they are suggesting measurements are accurate enough over 100 episodes, but it only applies to that problem – Neil Slater Sep 29 '18 at 17:53
• I have some problem about difference between episode and epoch! Episode=1 or k(k= multiple epoch) that after this k number of BP we have one episode? – user10296606 Sep 29 '18 at 18:06
• @user10296606: Sorry I don't understand your comment. There are not really epochs in RL (although you might decide to count some number of time steps as an epoch for a purpose such as viewing NN training metrics). You may need to ask a separate question and give more details about your specific problem – Neil Slater Sep 29 '18 at 18:52