Policy gradient on data only, without emulators

Question

It is too costly for my team to emulate the agent (executing the action and assessing the reward), meaning our only option is to learn the optimal policy on our dataset. The good thing is that we have a lot of data, that represents a sequence of state, action, reward. We can train our agent on this data.

We also need continuous actions, as the set of actions is big. Policy gradient is therefore the way to go but it generally uses an actor-critic that requires an emulator. We can not emulate, what would be the other options?

Answer 1

I think your best approach is to use Imitation Learning. Many techniques in imitation learning use Supervised Learning so you do not need to use emulator. Check DAGGER which is used in continuous action scenarios or the recent AggreVated algorithm (just ignore the theoretical parts of the paper).

As a start you can use Supervised Learning just for experimentation and then use the above algorithms. I would suggest though to use even a poor simulator just to have an idea of how your implementations behave after the. Bare in mind that RL tries to solve an optimization problem (maximize an utility/cost function) whereas the Supervised Learning methods try to optimize the difference between model's prediction and ground truth. Just be cautious of the algorithm's behavior before you go live.