1
$\begingroup$

I have a game environment I want to train an RL model on. This environment has 2 fundamental actions that the agent can take; "Left" or "Right" (say, 0 or 1).

However, the actions "Left" or "Right" can be taken in a discrete number of "degrees". For example, I can take action "Left" with degree 70% , or take action "Right" with degree 16%.

Assuming a discrete action space between 0-100% for each "Left" or "Right", making the total action space a discrete size of 201 (0-200 in increments of 1), does an agent learn the optimal degree to take either "Left" or "Right" in any given state?

$\endgroup$
1
$\begingroup$

If you are using a value-based method, like Q-learning in Deep Q Networks (DQN), then the "degree" concept has little meaning to the agent, and you are effectively training an agent to learn the best discrete action out of 200 actions.

Yes this could take a lot longer than learning a simple 2-action scenario, as the agent will not easily learn that the value of "Right 49%" is correlated to the value of "Right 52%" in any particular state.

This is a similar problem you might face in supervised learning if you wanted to move from classifying a dog vs cat image classifier to add bounding boxes for where the creature is. You don't add the bounding box as a large number of classifications - instead you add it as a regression, mapping form the image directly to some vector of real numbers.

There might be ways to address this using value-based methods, but the usual approach here would be to move to a continuous action space (maybe with -1 for 100% Right and +1 for 100% Left), and use a Policy Gradient method. There are a few different choices of Policy Gradient, but they all share the basic principle of modelling a policy function - which might be a probability distribution over action space which you need to sample from, or a specific action - converting from state to action.

The most basic Policy Gradient approach, which you could use as an introduction for your problem if it is simple enough, is REINFORCE. Algorithms like A3C, DDPG, PPO are based over this idea and can cope with more challenging environments.

$\endgroup$
  • $\begingroup$ Thanks for the explanation. So I take it policy gradient methods learn the best action even if the action space is continuous? $\endgroup$ – PyRsquared Jul 24 '19 at 15:06
  • $\begingroup$ @PyRsquared: Yes PG methods are basically required for continuous action spaces, as value-based methods suffer from problems defining the policy - finding $\text{max}_a \hat{q}(s,a)$ is the main problem. $\endgroup$ – Neil Slater Jul 24 '19 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.