Timeline for When should the last action be included in the state in reinforcement learning?
Current License: CC BY-SA 4.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 13 at 19:55 | comment | added | kyriakosSt | @johannesack I have to disagree with your last comment. Adding last action to state doesnt change the system's markov property at all. The relation between the past action and the part of the state vector that contains that action is trivially markovian, in that there is a trivial deterministic mapping. The rest of the state vector is unaffected by this addition. So if there was a prior markovian relation, then it still holds and vice versa. I agree with the rest of your comments. Also, the interesting question is whether the addition can help a (presumably neuralnet) agent learn faster. | |
| Mar 12, 2020 at 9:17 | comment | added | johannesack | @user91315 Essentially you have to ensure that your environment fulfills the Markov property. If you can do that by adding the last action to the state, go ahead. | |
| Mar 12, 2020 at 9:14 | comment | added | johannesack | @user91315 About the example where it would help: It can only help if the MDP is not really Markov, i.e. the last action gives some information about the transition function, that is not contained in the state. This might be because the state is poorly chosen. For example, in trading if you define your state as the current price of one stock but don't include whether you have bought it or sold it, you can not predict what reward you would get. So, in this case, you should add the action to your state. Then your state is your last action and the original state | |
| Mar 12, 2020 at 0:28 | comment | added | user91315 | I dont think my env is a proper MDP, its more like an POMDP (for example my env is similar to a financial trading env where the actions doesn't influence the price and there are many hidden factors that influence the price which are not observable) In any case, could you give an example of the rare case of POMDP where the last action would be included? @JohannesAck | |
| Mar 11, 2020 at 18:05 | history | answered | johannesack | CC BY-SA 4.0 |