1
$\begingroup$

Can Q Learning work with a static state for each step?

What I mean by that is that the actions do not influence the following state at all. The episodes just iterate over the same data over and over again. Of course the different actions lead to different rewards, but is Q Learning the right concept to use in this case? What other types could you suggest?

$\endgroup$
  • $\begingroup$ So to clarify: In your case, there is a fixed episode or cycle of states, always the same e.g. ${S_1, S_2, S_3 . . . S_N}$? You say that the reward depends on the action taken, but does it depend on the state too, so you are looking for best action to take in each state? What sort of length is the data, and do states repeat in it? How complex is each state? Do you need your learner to generalise to more than one set of fixed states? $\endgroup$ – Neil Slater Apr 5 '18 at 17:51
  • $\begingroup$ Another clarifying question that might help choose approach: Is this an online environment that requires an agent taking actions, and you want best reward whilst learning? Or are you trying to learn best actions from a simulator or historical data? $\endgroup$ – Neil Slater Apr 5 '18 at 17:54
1
$\begingroup$

The framework you describe is the Bandits framework. Bandits are algorithms that solve stateless MDPs. So the problem is not sequential in the sense that actions affect the transition to the next state. Take a look at a nice Contextual Bandits tutorial which is accompanied with Python Jupyter notebook. Please note that Contextual Bandits are a special category of Bandits but at least you will be exposed to the general Bandit idea and formalization.

You can try Q learning but with the Rescorla-Wagner rule $r(a) - Q(a)$ as the reward correction $\gamma\max_a' Q(s',a')$ does not make any sense as there is no transition to a subsequent state $s'$.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.