0
$\begingroup$

Context

I am confused about how a DQN is supposed to solve the cart pole problem since the rewards are so dense. I have been using pytorch example. I am aware of some solutions, but I have issue with the basic principle of the env.

Unlike the tutorials, I converted the state space representation into just the 1x4 returned state as opposed to an image. Also, I converted the action output to be a binned output. So the action 1x2 becomes 3x2 when binning is set to 3. So instead of getting the max action row-wise, I get a max action column-wise. I am using fixed targeting (training a primary and target DQN).

Question / Concern

My main issue with the env in general is that keeping the pole vertical is no different to the DQN as holding the pole to near-failure. How does the DQN get better if it is getting +1 reward regardless? My hypothesis is that keeping the pole tilted creates more samples in the memory. Then after, when optimizing the model, the tilted pole states get higher reward than the vertical pole states since there is a higher distribution of tilted pole states. How are we expected to expect a DQN to do well with this kind of reward set up? Wouldn't it be better to have Cart pole produce +1 reward only for if the pole is near vertical?

Extra Information

The goal here is to use the cart-pole for debugging a RL model, then shift it to multi-joint robot control.

The state is normalized to the expected min/maxes: Current State [[0.45564668 0.51196048 0.53126856 0.52450375]] The action input is Raw Action tensor([[0.9477, 0.9471]]) Bin Action [1.0, 0.0]. I am just using simplest action rep for testing below. I have also tested double dqn's, dueling dqns, and the use of PER. I have also been testing dropping the state space down to 1x1 via just inputting the angle of the pole.

X axis is the number of steps during an episode. Y axis is the number of episodes. enter image description here

enter image description here

$\endgroup$

1 Answer 1

1
$\begingroup$

You are mixing up two concepts from reinforcement learning, reward and return (aka utility)

  • Rewards are used to identify or specify goals of the agent. Whilst you can change them to help an agent focus on useful heuristics of the problem, it is more usual, especially in test/toy problems to have them very simple. In the case of CartPole, there is a positive reward for "not falling over" which importantly ends when the episode ends.

  • Returns (or utility) are what the agent learns to maximise. A return is typically the sum of all rewards, and might be discounted to prevent infinite results. In the case of CartPole, this means that the longer the agent can balance the pole into the future, the larger the return is.

With Q-learning, the action values predict the expected future return. So it doesn't matter that the rewards are dense. It matters how long the agent can keep balance going into the long term, the longer the better, because the return will be higher. A combination of state and actions that the agent associates with longer-lasting not failing will predict a larger return and be chosen in preference to shorter-term success. This is how the Q-learning agent handles a situation with dense positive rewards that may end on a mistake.


In practice, the OpenAI Gym CartPoleV0 environment does take a small liberty. Episodes will end at a fixed step in future. This is not available in the state information, and technically makes the problem non-Markov. However, it is possible to get away with this provided the timespan for maintaining stability is shorter than the maximum possible episode.

$\endgroup$
1
  • $\begingroup$ I think I understand this better. My implementation was not working due to the random exploration ending too early, and my memory being too large. So based on what you said, that means that if my DQN stays "tilted" too often, then will be eventually get stuck there. Extending the random exploration solved this for me because there were more opportunities for the pole to stay straight up. There appears to be a balance between memory size and random exploration that I needed to strike. $\endgroup$
    – Josiah L.
    May 10, 2019 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.