2
$\begingroup$

In Q-learning, how to tell the agent that action $a_7$ is unavailable from within state $s_{t}$?

Is supplying a very large negative reward good, or might throw it off-track? From what I read (link), it's best to work with small rewards (between -1.0 and 1.0)

$\endgroup$
3
$\begingroup$

In Q-learning, how to tell the agent that action $a_7$ is unavailable from within state $s_{t}$?

Partially, it depends on what you mean by "unavailable". If the environment is such that it is possible to take the action, but that the consequences are very bad, then a negative reward due to the consequences is probably the best option. You might do this for a real-time system where the action can actually be taken, cannot be automatically blocked in any way, and has unwanted consequences.

Perhaps an action is unavailable, because it is not possible to even attempt it in the supplied environment. This could occur in a board game where certain moves are not allowed by the rules of the game. There you have a few options, depending on how you have constructed the agent.

A simple approach is to only present the agent with actions that it is allowed to take in the first place. The code for the environment should already know this list, so if you were considering penalising the choice, then you can also choose not to present it. This works well for value-based methods such as Q-learning, when you have implemented $Q(s,a)$. If the set of allowed actions is $\mathcal{A}(s)$, then the greedy action choice from state $s_t$ is $a_t = \text{argmax}_{a \in \mathcal{A}(s_t)} Q(s_t, a)$. All you need to do is implement $\mathcal{A}(s)$ in code and loop over different calls to Q(s,a) to find the best value.

If you have implemented the Q function in parallel over the whole action space (a good choice for efficiency with neural networks), then the network takes only the state $s$ as input, and outputs a vector e.g. $[Q(s, a_0), Q(s, a_1), Q(s, a_2), Q(s, a_4), Q(s, a_5) ... Q(s, a_N)]$. In that case you would use your implementation of $\mathcal{A}(s)$ to mask out unwanted actions before choosing the best action. It is a little bit wasteful to calculate the non-required actions, but if it was a rare enough exception, then probably still more efficient overall.

Beyond Q-learning, you could also use a similar action mask when you had a policy network that output softmax probabilities of actions. In that case you would also want to re-normalise the probabilities before generating the selected action.

Unless you have a specific goal or benefit in mind for the agent to learn about actions that are blocked by the environment, then it is simplest to query the environment to support the blocking, as described above. This is definitely the case for game-playing bots covering moves which are illegal according to the game rules. The justification is that the agent is learning to play the game, and the goal of it learning what the rules are for the game is a less interesting distraction.

Alternatively, it should work to allow the incorrect action selection, then give a negative reward (can be a small one, in same scale as other rewards in the system). Note this should also increment the time step and return the unaltered state. An agent will eventually learn not to take such a pointless, barred action. Maybe sometimes you are interested in whether the agent can learn the rules.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.