4
$\begingroup$

My question is about how to monitor RL agents in production. To make the question easier to discuss, here is a use case. Please don't focus on difficulties in implementing such an agent, but rather on how to monitor if it is still doing well:

Suppose Amazon would use Reinforcement Learning for optimizing the order of a search:

  • An episode starts when a user starts a search
  • An episode ends after a threshold or when the user buys things.
  • The observation the agent gets is the search terms as well as 20 products for which he has to give an ordering.
  • He gets a reward if one of those 20 things is bought.

Of course, one can (should?) let the agent learn all the time as products change and probably search terms / language changes. But when do I know that the agent learned something weird / that I should stop it?

I can imagine the following:

  • Case-based single examples
  • Having a ground-truth for some trivial searches + products and checking if the model does those right (in non-exploratory mode)
  • Letting the agent learn in batch-mode (e.g. update the model only once a week) and doing A/B tests for the current model / new model.
  • Measuring mean reward and setting a threshold. If the mean reward of the agent drops below the threshold, reset the agent to a past "save" state.

Is there literature about it? Blog posts?

I know at least one example where RL went wrong / monitoring didn't quite work:

$\endgroup$
2
+50
$\begingroup$

This is an open issue in Reinforcement Learning (and all of machine learning). Google has published a paper titled "Machine Learning: The High Interest Credit Card of Technical Debt" which addresses the many ways these systems can degrade over time.

One way is to follow fundamental production engineering techniques (e.g., test coverage and graceful exit).

Another way is to monitor the data, then retrain the agent if new data is statistically out-of-sample.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.