4
$\begingroup$

I'm familiar with traditional reinforcement learning where the algorithm must choose a categorical action (e.g., best move in a game or the highest click-through-rate ad among a set of ads). Does reinforcement learning work for problems with continuous actions? Are there any easy to understand references that you recommend? I was hoping to be able to use reinforcement learning to assign credit limits, which are represented in dollars (i.e., continuous amounts).

Edit: I've come across this paper from Google's DeepMind, which so far seems relevant.

$\endgroup$
5
  • $\begingroup$ Technically assigning a credit limit would be an action, not an outcome. But yes you can have continuous actions and states. Usually there is more than one action to consider, a series of states connected by those actions, and a reward scheme. Is that really going to happen for your problem, or is it more in the ballpark of predicting outcomes from single events? $\endgroup$ – Neil Slater Jul 18 '17 at 19:34
  • $\begingroup$ @NeilSlater Good point. I've updated the question title to use the word action instead of outcome. Think of a credit limit as like a loan amount. A reckless debtor says, "Give me the biggest loan possible!" and the banker has to says, "I feel comfortable giving you a loan of $X." So there is only one type of action: setting the loan amount (in my case, credit limit). Rewards are interest and fees and penalties are delinquencies or charge-offs (ie amount owed that is never collected, for example because of debtor bankruptcy). $\endgroup$ – Ryan Zotti Jul 18 '17 at 19:42
  • $\begingroup$ My particular problem is well suited for RL because the "loan" is actually more like a daily revolving line of credit that can be raised or lowered over time as my employer collects feedback on the debtor's payment performance. $\endgroup$ – Ryan Zotti Jul 18 '17 at 19:44
  • $\begingroup$ The deep actor-critic thing in the paper looks good to me. I'm still slugging my way through Sutton & Barto's book, and avoiding jumping in at that level, but I expect if you find an implementation you could give it a go. The main problem you face is testing it. You can use your historical data to train the agent so it can start with an approximation to your employer's current policy and value function. But at some point you have to give it control, and you don't have a simulated environment with well-modelled debtors in which to allow it to play safely - it would have to be real! $\endgroup$ – Neil Slater Jul 18 '17 at 19:56
  • $\begingroup$ Yeah, my employer and I will definitely have to pay careful attention. I would probably only use it on a small random sample of customers and see how the RL profit/loss rates compare to the control (our existing process). Fortunately, due to how the product is designed, we get feedback much faster compared to other financial institutions (days vs years), so I can quickly correct course if necessary. It's a very interesting project. $\endgroup$ – Ryan Zotti Jul 18 '17 at 20:11
2
$\begingroup$

The DDPG described in that paper is very good to start working in control problems. There are many implementations of this algorithm online. However, bare in mind though that the algorithm is very unstable and needs many hours of monitoring and tuning the parameters of the networks. I would suggest you to create a very simple scenario (simulator) and put the DDPG to interact with it. It is very important to understand what your networks are learning and which state features are important or no. Furthermore the reward function will play a very important role in the performance of your algorithm.

Another important point is the way you formalize your problem will determine if you are trying to solve a one-step decision making problem (e.g. given state features ---> credit limit value, reward) or a sequential decision making problem (e.g. given state features ---> credit limit value, reward, new state features and so on). This will specify if you need to modify the Q learning part of the Critic and remove the $maxQ(s',a')$ part (one step decision case).

Regarding pretraining, as Neil suggested you can use regression to pretrain Actor and Critic networks so the weights are initialized to better than random values. A very interesting approach from Deepmind is the one that is described in this paper Learning from Demonstrations for real-world RL, and most likely helped them optimize the energy consumption at their server centers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.