2
$\begingroup$

I have a fundamental question on the applicability of reinforcement learning (RL) on a problem we are trying to solve.

We are trying to use RL for inventory management - where the demand is entirely random (it probably has a pattern in real life but for now let us assume that we have been forced to treat as purely random).

As I understand, RL can help learn how to play a game (say chess) or help a robot learn to walk. But all games have rules and so does the ‘cart-pole’ (of OpenAI Gym) – there are rules of ‘physics’ that govern when the cart-pole will tip and fall over.

For our problem there are no rules – the environment changes randomly (demand made for the product).

Is RL really applicable to such situations?

If it does - then what will improve the performance?

Further details: - The only two stimuli available from the ‘environment’ are the currently available level of product 'X' and the current demand 'Y' - And the ‘action’ is binary - do I order a quantity 'Q' to refill or do I not (discrete action space). - We are using DQN and an Adam optimizer.

Our results are poor - I admit I have trained only for about 5,000 or 10,000 - should I let it train on for days because it is a random environment?

thank you Rajesh

$\endgroup$
3
  • $\begingroup$ Where do you train your RL algorithm? Do you use already collected data for this and if yes can you give a brief description of a cycle (obs, step, action, obs')? Also trying to map one variable to expected reward (Q function) using NNs is not a good strategy, especially in a volatile scenario as the one you describe. $\endgroup$ Oct 18 '18 at 23:06
  • $\begingroup$ Dear Constantinos: We do not use collected data - but randomly generated 'demands' (they do lie in a range of say 0 to 1000 units) >> Also trying to map one variable to expected reward (Q function) using NNs is not a good strategy What strategy would then be suitable? $\endgroup$
    – RajeshS
    Oct 29 '18 at 15:33
  • $\begingroup$ So actually you are saying that you try to map the currently available product X with a random number from 0 to 1000 right? Or these 'demands' are random samples of a particular probability distribution? For now I am thinking that your problem is not formulated appropriately for solving with RL but better with SL...although if you try to map a quantity X to random numbers no matter the method it won't lead you anywhere (am sure you know that already). $\endgroup$ Oct 30 '18 at 6:33
3
$\begingroup$

Taking your question title literally:

Is RL applicable to environments that are totally RANDOM?

The answer would be no. In a totally random environment there is little to nothing that can be learned.

However, you don't actually have a totally random environment. You have some quantities that fluctuate a lot in a way that you don't understand. Otherwise your environment behaves very logically - if there is demand for Y items, and you have X in stock, then you will end up with X - Y in stock if X > Y, or 0 in stock and Y - X incomplete orders otherwise. This is a very structured rule that you can definitely associate rewards with and learn. Presumably you have costs for ordering and holding stock and opportunity costs for not supplying orders.

A simple variant of your situation is used as a toy example in Sutton & Barto: Reinforcement Learning, an Introduction in chapter 4, called "Jack’s Car Rental" where the goal is to optimise stock at two locations with random demand occurring at either location. This toy problem is made easier by defining the distribution, and using it in a model-based way. But that is not necessary in general.

Is RL really applicable to such situations?

In your case, probably yes. Although you do have to assume that "totally random" is just a way of phrasing "highly variable" and not literally 0 one day, 30 million the next day and 7.5 the next day. There are going to be limits to the orders and they will follow some distribution.

With very high variance then you are likely to find it hard to reach a balance point of costs for holding stock vs costs for missing orders, but in principle this is solvable and RL is a reasonable tool to attempt the solution.

If it does - then what will improve the performance?

Check whether any fluctuations in demand depend on any variables that you can collect and add those variables to the state. For instance, if demand has some weekly or seasonal variation, then those parts of the date should be part of your state representation.

Understanding and predicting the distribution of demand, even if not the exact values, would also help with simulations and planning algorithms. The RL algorithm will over time learn the likely distribution, but it can only base its own predictions on state variables that you have let it observe.

You might be able to go one better: If demand is mostly independent of the stock levels that you hold, then you can separate the problem of predicting it and use more robust supervised learning to add a "predicted demand" feature to the state. The RL will on top of this prediction learn the costs associated with just trusting this prediction vs allowing for some extra incoming orders just in case etc.

$\endgroup$
2
  • $\begingroup$ Dear Neil, Thank you for a detailed reply. I just tried a LOT of things after being encouraged by your post. Question: - The rewards are incremented using a rewards function so that they increase exponentially as each demand in the day is met - I thought that this will help 'nudge' the agent to meet more and more of the demand without running out of stock or overflowing the shelf (the two 'done' or 'stop' conditions) My Q values are NEGATIVE for both actions after about say 1,000 episodes and huge e.g. [-40236876. -94196420.] Is this OK? Can they be so large and -ve? $\endgroup$
    – RajeshS
    Oct 29 '18 at 15:46
  • $\begingroup$ Exponentially increasing rewards seems wrong to me - especially if the agent does not have any detail in the state that gives it a clue that this might happen. You should not need to "help" an agent, instead you should give it consistent rewards that make some kind of measurement sense in the context of the problem. E.g. often for stock problems, the expected long-term monetary value. Your large negative Q values don't seem successful to me, but it is hard to tell without knowing the full details of your problem. They could be caused by several types of error. $\endgroup$ Oct 30 '18 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.