My question is about how to monitor RL agents in production. To make the question easier to discuss, here is a use case. Please don't focus on difficulties in implementing such an agent, but rather on how to monitor if it is still doing well:
Suppose Amazon would use Reinforcement Learning for optimizing the order of a search:
- An episode starts when a user starts a search
- An episode ends after a threshold or when the user buys things.
- The observation the agent gets is the search terms as well as 20 products for which he has to give an ordering.
- He gets a reward if one of those 20 things is bought.
Of course, one can (should?) let the agent learn all the time as products change and probably search terms / language changes. But when do I know that the agent learned something weird / that I should stop it?
I can imagine the following:
- Case-based single examples
- Having a ground-truth for some trivial searches + products and checking if the model does those right (in non-exploratory mode)
- Letting the agent learn in batch-mode (e.g. update the model only once a week) and doing A/B tests for the current model / new model.
- Measuring mean reward and setting a threshold. If the mean reward of the agent drops below the threshold, reset the agent to a past "save" state.
Is there literature about it? Blog posts?
I know at least one example where RL went wrong / monitoring didn't quite work: