0
$\begingroup$

Say we have a game that is a maze environment where there is a character to be controlled through the maze. When the agent (the character) approaches a wall, it may try to execute an action that would lead it into the wall, which is not permitted. Instead of executing this action, we place an if statement in the code that checks if the action is permitted, and if not, does not execute the action, and simply proceeds to the next state.

Another similar example is if an RL agent is being trained in a stock trading environment, where say it tries to sell more stock than it actually owns, or buy less than the minimum amount. Similarly as before, we place an if statement that checks for these conditions and either allows the action (allows the trade) or moves on to the next state.

Does the agent still learn, even if we "override" it and block certain actions? How else would we go about restricting the agent to certain actions in certain states e.g. if a wall is to the left of the agent in the game environment, how would we restrict the agent to only move forward, backward or right, but not left?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

By overriding the Agent's action, Agent can theoretically take this action over and over and do nothing else, there is nothing to stop the agent from doing this, or teach the agent that this is not the intended action here.

What I usually do in these situations is, I punish the agent if the action is not desirable. So if agent take the action to go to the left, but there is a wall on the left side, I would not change the state of the environment (So the agent won't move), but I would also send a minus value (punishment) as a reward. This way after some training, the agent would learn that this action is not desirable.

The same can be applied to you Stock example. So if the agent tries to sell more stock that it actually has, you just don't sell the stock but also punish it with a big minus reward. This way it makes it easier for the agent to actually understand the environment.

Hope this helps.

Improve this answer
$\endgroup$
2
  • $\begingroup$ That makes intuitive sense. Thanks! $\endgroup$
    – PyRsquared
    Commented Jul 30, 2019 at 8:07
  • 1
    $\begingroup$ I would add that it is usually not necessary to add such a negative reward, if there is already a consequence for using a time step. E.g. in a maze, the reward is typically -1 per time step. Taking an action that causes no movement will quickly be leanred away with no special handling. The stock trading example might be different as there is presumably a deliberate "wait" action, and you don't want the agent to force itself to wait by proxy of choosing non-valid actions that may in slightly different circumstances become valid and cause extreme behaviour (like selling all stock). $\endgroup$
    – Neil Slater
    Commented Jul 30, 2019 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.