When should the last action be included in the state in reinforcement learning?

I am having some confusion as to whether the action should be included as part of the state input to an agent in a reinforcement learning setting (state-action pair). As from my observation, this is not completely clear as different agents/environments combinations might have different performances if action was included/excluded from input states (I might be wrong).

For my specific problem:

• the agent can't influence/control the states through its actions (similar to the case of a simple multi-armed bandit)
• the action space is discrete
• I am using a DQN based approach

I would also appreciate a general overview/rules of thumb of when to include/exclude actions as state inputs.

ps. when i say "different agents/environments combinations" in the beginning I mean using different agents to solve the same env or same agent to solve different env.

1 Answer

If your environment fulfills the Markov property, there is no reason to include the actions $$a$$ in the state $$s$$, as the action $$a_t$$ that lead to the new state $$s_{t+1}$$ should not provide any additional information, i.e. the knowledge of the old action should not influence the reward and transition functions from $$s_{t+1}$$ onwards.

Therefore, if your environment is a proper MDP, there is no reason to include actions in the next state.

It might be helpful in some rare cases, where your environment is not actually Markov to include the last action in your state, but in that case you should also think about if there is a better way to represent your state, such that it becomes Markov.

In your concrete case, where the actions do not change the state, there is no reason to include the last action as part of your state.

Side Note: If your problem is a simple discrete bandit, you will probably be better off by looking into bandit approaches, rather than using a DQN based approach.

• I dont think my env is a proper MDP, its more like an POMDP (for example my env is similar to a financial trading env where the actions doesn't influence the price and there are many hidden factors that influence the price which are not observable) In any case, could you give an example of the rare case of POMDP where the last action would be included? @JohannesAck Mar 12 '20 at 0:28
• @user91315 About the example where it would help: It can only help if the MDP is not really Markov, i.e. the last action gives some information about the transition function, that is not contained in the state. This might be because the state is poorly chosen. For example, in trading if you define your state as the current price of one stock but don't include whether you have bought it or sold it, you can not predict what reward you would get. So, in this case, you should add the action to your state. Then your state is your last action and the original state Mar 12 '20 at 9:14
• @user91315 Essentially you have to ensure that your environment fulfills the Markov property. If you can do that by adding the last action to the state, go ahead. Mar 12 '20 at 9:17