11
$\begingroup$

I'm trying to solve OpenAI Gym's LunarLander-v2.

I'm using the Deep Q-Learning algorithm. I have tried various hyperparameters, but I can't get a good score.

Generally the loss decreases over many episodes but the reward doesn't improve much.enter image description here

How should I interpret this? If a lower loss means more accurate predictions of value, naively I would have expected the agent to take more high-reward actions.

Could this be a sign of the agent not having explored enough, of being stuck in a local minimum?

$\endgroup$
4
  • $\begingroup$ Out of interest, are you using a discount factor? If so, what value? I have noticed something about it that may be relevant. $\endgroup$ Sep 12, 2018 at 19:15
  • $\begingroup$ @NeilSlater I had tried various discount factors, settling for 0.99 $\endgroup$
    – Atuos
    Sep 13, 2018 at 8:09
  • $\begingroup$ I was thinking 0.99 is probably too low, due to number of time steps between start and potential crash vs landing. However, I am not getting good results in my implementation (which works a charm on CartPolev1). Playing with hyperparameters, especially exploration rate, I can get sequences of successful landing, but the agent easily forgets them. I'm still exploring this, it is an interesting problem to find out why basic DQN learning struggles here. $\endgroup$ Sep 13, 2018 at 8:26
  • 1
    $\begingroup$ @NeilSlater I don't know, but you may be right. It may be helpful to think about how many steps it takes for the discounted reward to be halved. With 0.99 it is ln(0.5)/ln(0.99) = 68 steps. Episodes typically last 200-1000 steps. Have you tried decaying your learning rate? Maybe having the learning rate decay once it has completed a few landings would prevent it from going off the rails again. $\endgroup$
    – Atuos
    Sep 13, 2018 at 10:57

1 Answer 1

10
$\begingroup$

How should I interpret this? If a lower loss means more accurate predictions of value, naively I would have expected the agent to take more high-reward actions.

A lower loss means more accurate predictions of value for the current policy (technically it is more complicated for Q-learning off-policy estimates, but the covergence will still be limited by experience reachable in the current policy). Unfortunately a loss metric in RL cannot capture how good that policy is.

So what it means is that your policy has settled into a pattern where values can be estimated well by the neural network that you are using for Q. For some reason it is not finding improvements to that policy - typically it should be doing that before the loss metric drops, as each improvement in value estimates should reveal better possible actions, and once those start being taken by a new policy, the value estimates become out of date, and the loss increases again.

Could this be a sign of the agent not having explored enough, of being stuck in a local minimum?

Exploration could be an issue. The "local minimum" in that case is probably not an issue with the neural network, but that small variations in policy are all worse than the current policy. As you are learning off-policy, then increasing the exploration rate may help find the better states, at the expense of slower overall learning. Also, methods that explore more widely than randomly on each action could be better - e.g. action selection methods that consistently pick unexplored state/action pairs such as Upper Confidence Bound.

Also a possibility is that the structure of your network generalises well under the current policy, but is not able to cover better policies. In that case, whenever the exploration suggests a better policy, the network will also increase estimates of unrelated action choices - so it would try them, notice they are better, then back off as the new values also cause unwanted policy changes in other situations.

If you know a better policy than the one that is being found, then you could plot a learning curve with the policy fixed, see if the network can learn it. However, usually you will not know this, so you may be stuck with trying some variations of neural network architecture or other hyperparameters.

There are other methods than DQN (e.g. A3C, DDPG), as well as many add-ons and adjustments to DQN that you could try (e.g. eligibility traces, double learning).

$\endgroup$
4
  • $\begingroup$ This is an outstanding answer and very well written. Thank you very much. I think I will be trying double learning next. As an aside, the pattern I see is this: my agent tries to land the craft many times, but usually fails, losing lots of points. It then gives up on landing, realizing that it loses less points if it just hovers until the episode ends. Sometimes it tries new stuff with little success, and reverts back to hovering. $\endgroup$
    – Atuos
    Sep 4, 2018 at 13:10
  • 3
    $\begingroup$ @Atuos: Actually I realised a mistake and corrected it - Q-learning tries to learn optimal Q values from a current non-optimal policy, and to some degree it doesn't matter what the current policy is. However, until your agent can experience at least one successful landing, it will never be able to learn that is a good thing, and the Q estimates will reflect that (and can become stable if all the history shows the same). I don't think double-learning will help you here $\endgroup$ Sep 4, 2018 at 13:32
  • 2
    $\begingroup$ @Atuos: I am replicating your problem using single-step Q Learning. Having a high random exploration for a long period does seem to help, but I suspect that something different or extra is required to make Q learning reliable. It looks to me that the reward scheme - which is very sparse, and very hard to get the top reward by chance - is designed to create this issue as a challenge to learning agents. $\endgroup$ Sep 12, 2018 at 7:26
  • $\begingroup$ Yes, this seems like a difficult problem for Q-learning, though inexplicably using a vanilla DQN agent I found on GitHub produced very good results. Even with the same hyperparameters as my implementation, the agent converges quickly and produces great scores. I have read the code but am still at a loss for how to explain it. github.com/keras-rl/keras-rl/blob/master/rl/agents/dqn.py $\endgroup$
    – Atuos
    Sep 13, 2018 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.