if we train a DQN over 40000-60000 episodes for 500 time steps. The mean of reward during last 100 training steps is about 1.1 times of reward during the test process.

Environment is stochastic!

The more time steps should be more stable?? What is the effect of time steps?

For instance, the mean of reward during the last 100 episodes of training is 6.1 however the answer during the test is 5.2 (My problem is minimization). Could we say our DQN is OK? One test random is 120! however another is 400!

Or vice versa, is it acceptable if the reward of test in minimization is more than for instance 2 times of mean of last 100 training set? Could we say our DQN is OK?

I did not run as a human! but if it is important I will play this game.

  • $\begingroup$ That is too little data to say one thing or another. Please provide more details about your environment, what the reward represents. What does a completely random agent get for reward? What does a human get on the same task? Is the environment deterministic or stochastic. How many test steps did you take? What do you mean by "steps" - individual time steps or complete episodes? $\endgroup$ – Neil Slater Oct 14 '18 at 7:51
  • $\begingroup$ Neil Slater , I added more details $\endgroup$ – user10296606 Oct 14 '18 at 15:48
  • $\begingroup$ Thank you for adding details. I am afraid the question is still quite hard to understand. However, I have done my best to answer, and I hope that it helps. $\endgroup$ – Neil Slater Oct 14 '18 at 17:35

There is a lot of information that might be useful for a more precise answer in your case, but is hard to extract from your question description.

I think I can understand the following:

  • You are training to minimise a cost instead of maximise a reward (this is unusual in RL, but not a problem).
  • Random agents score variably, anything from 120 to 400 average cost.
  • After many training steps, your agent is scoring 6.1 average cost, measured over the most recent 100 time steps.
  • In testing, the agent scores better, an average cost of 5.2.

The usual way of testing a DQN agent (or any off-policy agent, of which Q Learning or DQN are one type) is to stop exploring actions. I am assuming you do this too. During training, your agent was not always taking the actions that it determined were best. In tests, you are interested to see what the agent thinks is "best" and how good that is. So it should be expected that test scores better than recent training. If you were training using $\epsilon$-greedy action selection, then your scores would be consistent with $\epsilon \approx 0.1$

However, your measurements have limited accuracy in any case. It might be worth calculating your standard error on your test data, to get a sense of how accurate your measure really is. The easiest way is to run your test multiple times, create an array of scores and calculate the standard deviation, mean and standard error as usual using the test results as samples of a random variable.

Could we say our DQN is OK?

That depends entirely on the problem. It seems that it does far better than acting randomly, which shows at least it has learned something.

In order to get a sense of how well it has done, you need to compare it with something meaningful related to the problem:

  • For simple problems, such as toy problems used to prove DQN works, it is possible to calculate an optimal answer and see how close the DQN can get.

  • For problems that humans can attempt too, you can compare the performance of an agent to a human. Ideally a human who is expert at the task.

  • For problems for which there are some publications, you can compare with state-of-the-art. This is true for many Atari 2600 games for instance.

  • For problems which have a practical purpose, you can compare with a goal, such as would the agent be good enough to make money or reduce loss compared to current practice.

  • For some problems you could compare to a simple rules-based agent, coded to perform the specific task.

The more time steps should be more stable?? What is the effect of time steps?

Up to a limit, the more time and samples that a DQN trains on, the better its performance will be. The best thing is to plot this - number of timesteps vs average cost. You may see sudden increases or even "catastrophic forgetting" (where the agent does no better than random, or even worse) at some times. But if the agent has trained successfully there should be a general improvement over time that reaches a limit as the DQN approaches the maximum capability of the agent.

  • 1
    $\begingroup$ Great answer, I would add another approach of implementing a simple rules based heuristic agent that will likely do a lot better than random. $\endgroup$ – Jan van der Vegt Oct 14 '18 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.