2
$\begingroup$

My team and I started digging into RL for the purpose of a specific application. We have plenty of data of an agent carrying out suboptimal policies (states and rewards...).

It is too costly for us to emulate the agent (executing the action and assessing the reward) meaning our only option is to learn the optimal policy on our dataset using an off-policy algorithm. However I still do not know how to do that, Q-learning, while off-policy, still require an emulator.

Could you please provide me some guideline on how to do that? Which type of algorithm should I use?

$\endgroup$
2
$\begingroup$

Q-learning does not in fact need to be online or need an emulator, it can learn exclusively from experience replay. If you put all your history into a table or state, action, reward, next state and then sample from it, it should be possible to train your agent that way.

To do this, you will need to skip the algorithm steps that take actions and store results. The algorithm will then learn from the data you have. It will just not be possible to collect more. Depending on the problem you are trying to solve, this could be OK, or it may inhibit learning.

RL algorithms learning optimal control in complex environments benefit from sampling near to their current policy, so it is possible in your case that your agent will reach a limit on what it can learn from historic data. It may end up still quite far from optimal behaviour, although it should stand a reasonable chance of improving on the best that the historic data shows.

If you need to use function approximation (e.g. a neural network) due to size of the state, action space, then take extra care, because it will be hard to detect whether the action values have converged correctly. This is because you are learning the optimal Q values, and you will have no test data that demonstrates what those should be (to collect that data, you need to follow the optimal policy and measure the total reward).

Here is roughly what an experience-replay-only Q-learning algorithm would look like:

Input: History $H$, consisting of rows of $S,A,R,S'$

Initialise the NN for calculating $\hat{q}(s,a)$

Repeat until NN converges:

$\qquad$ Sample $S,A,R,S'$ from $H$

$\qquad$ $tdtarget \leftarrow R + \gamma \text{max}_{a'}[\hat{q}(S',a')]$

$\qquad$ Train NN single step, $\hat{q}(S,A) \rightarrow tdtarget$

You can make use of mini-batch processing to generate multiple $tdtarget$ values at once and train on them. A worthwhile improvement for stability is to use a frozen copy of the neural network when calculating the $tdtarget$ value, and update it only every N steps to be a copy of most recent network, with N maybe 1000 steps.

$\endgroup$
  • $\begingroup$ Thanks for your detailed answered. Indeed I need to use function approximation/NN. Could you clarify how to use minibatch? Do you mean I need to store the data in the same states (among my episodes) and sample from this data. Also, sorry I do not understand the frozen copy of the neural network. $\endgroup$ – user26616 Feb 3 '18 at 15:09
  • $\begingroup$ @user26616: minibatch just means running multiple samples at once on each learning step. So just take 20 samples, create 20 $tdtarget$ values, and train on the combined gradients. It is for efficiency, and most NN frameworks will support processing in batches of examples at a time. Frozen copy = a copy that you do not train. $\endgroup$ – Neil Slater Feb 3 '18 at 15:55
  • $\begingroup$ Thanks a lot. I suppose for this type of application off-policy Monte Carlo is a better algorithm. What is the best algorithm? $\endgroup$ – user26616 Feb 7 '18 at 7:14
  • $\begingroup$ @user26616 I don't know. As with supervised learning, usually the best algorithm is determined by experiment on a problem, there is not much you can say a priori. Knowing the size of state and action spaces might help. The biggest problem with off-policy Monte Carlo is that to apply importance sampling you need to know the action probabilities under the behaviour policy, and you may not have that data. $\endgroup$ – Neil Slater Feb 7 '18 at 8:11
  • $\begingroup$ I made a q-learning algo running on a small set of actions. One thing that I have to deal with now is the fact that my set of actions is very large and I am better off modeling it continuously. Therefore, I need to use a policy gradient algo. What are your suggestions in terms of algos I could use given the original specifications? $\endgroup$ – user26616 Apr 11 '18 at 17:28

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.