I am having some confusion as to whether the action should be included as part of the state input to an agent in a reinforcement learning setting (state-action pair). As from my observation, this is not completely clear as different agents/environments combinations might have different performances if action was included/excluded from input states (I might be wrong).

For my specific problem:

  • the agent can't influence/control the states through its actions (similar to the case of a simple multi-armed bandit)
  • the action space is discrete
  • I am using a DQN based approach

I would also appreciate a general overview/rules of thumb of when to include/exclude actions as state inputs.

ps. when i say "different agents/environments combinations" in the beginning I mean using different agents to solve the same env or same agent to solve different env.


1 Answer 1


If your environment fulfills the Markov property, there is no reason to include the actions $a$ in the state $s$, as the action $a_t$ that lead to the new state $s_{t+1}$ should not provide any additional information, i.e. the knowledge of the old action should not influence the reward and transition functions from $s_{t+1}$ onwards.

Therefore, if your environment is a proper MDP, there is no reason to include actions in the next state.

It might be helpful in some rare cases, where your environment is not actually Markov to include the last action in your state, but in that case you should also think about if there is a better way to represent your state, such that it becomes Markov.

In your concrete case, where the actions do not change the state, there is no reason to include the last action as part of your state.

Side Note: If your problem is a simple discrete bandit, you will probably be better off by looking into bandit approaches, rather than using a DQN based approach.

  • $\begingroup$ I dont think my env is a proper MDP, its more like an POMDP (for example my env is similar to a financial trading env where the actions doesn't influence the price and there are many hidden factors that influence the price which are not observable) In any case, could you give an example of the rare case of POMDP where the last action would be included? @JohannesAck $\endgroup$
    – user91315
    Mar 12, 2020 at 0:28
  • $\begingroup$ @user91315 About the example where it would help: It can only help if the MDP is not really Markov, i.e. the last action gives some information about the transition function, that is not contained in the state. This might be because the state is poorly chosen. For example, in trading if you define your state as the current price of one stock but don't include whether you have bought it or sold it, you can not predict what reward you would get. So, in this case, you should add the action to your state. Then your state is your last action and the original state $\endgroup$ Mar 12, 2020 at 9:14
  • $\begingroup$ @user91315 Essentially you have to ensure that your environment fulfills the Markov property. If you can do that by adding the last action to the state, go ahead. $\endgroup$ Mar 12, 2020 at 9:17
  • $\begingroup$ @johannesack I have to disagree with your last comment. Adding last action to state doesnt change the system's markov property at all. The relation between the past action and the part of the state vector that contains that action is trivially markovian, in that there is a trivial deterministic mapping. The rest of the state vector is unaffected by this addition. So if there was a prior markovian relation, then it still holds and vice versa. I agree with the rest of your comments. Also, the interesting question is whether the addition can help a (presumably neuralnet) agent learn faster. $\endgroup$
    – kyriakosSt
    Feb 13 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.