1
$\begingroup$

If you train an agent using reinforcement learning (with Q-function in this case), should you give a negative reward (punish) if the agent proposes illegal actions for the presented state?

I guess over time if you only select from between the legal actions, the illegal ones would eventually drop out, but would punishing them cause them to drop out sooner and possibly cause the agent to explore more possible legal actions sooner?

To expand on this further; say you're training an autonomous vehicle, and the output is drive direction (forward or reverse) and speed. Say for the scenario you're in, the vehicle must drive between a speed range, e.g. 20mph min, 40mph max, what do you do in the scenario where the agent gives an action to drive forward but gives a speed below the minimum speed? Or another example, say you're training to play a game, and the agent proposes an illegal action which it cannot perform.

I can't proceed with the action because it's illegal, so what do I do? How do I proceed with training in that situation? I will of course enter the min/max speeds as part of the state given to the agent, but how do I prevent it from proposing actions that are illegal, and how do I proceed with training when it does?

$\endgroup$
1

1 Answer 1

0
$\begingroup$

I think that you should specify better what an illegal action is. Suppose to me in a highway with a self-driving car. I don't which are your legal limits values but suppose that they are between 20mph and 40mph. Of course, the car itself could drive at less than 20mph and let us suppose that its maximum possible velocity is 60mph. If your self-driving car is driving at less of 20mph or between 40mph and 60mph you should give a negative reward for each time-step it is outside legal limits. Instead, if your self-driving car is at more than 60mph you have a problem in your environment because 60mph is a physical limitation and should be handled by the environment. An easy solution would be to clip the action between 0mph and 60mph and give negative reward if it's in the range 0mph to 20mph or if it is in the range 40mph to 60mph.

$\endgroup$
3
  • $\begingroup$ An illegal action being one it cannot do at any time. Assume it is literally not possible to do the illegal action at all, for example, in a game it would contravene the rules. $\endgroup$
    – BigBadMe
    Dec 2, 2018 at 12:05
  • $\begingroup$ @BigBadMe. Then, it should be handled by the environment. $\endgroup$
    – gvgramazio
    Dec 2, 2018 at 12:42
  • $\begingroup$ @BigBadMe. It all depends of the rules. I'm able to kill a person but then I will go to jail because it's not allowed by my state's rules .I'm not able to revive a person because it's not allowed by the rules of nature. The environment is both the state and the nature: it will allow you to kill a person but then it will send you to jail but it will not allow you to revive a person even if it's what you want to do. Personally, I will define the first a bad action (thus the negative reward) and the latter an illegal action (thus the prohibition to take it). $\endgroup$
    – gvgramazio
    Dec 2, 2018 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.