Handling actions with delayed effect (Reinforcement learning)

I am working on a problem where the action that I learn (using DQN) can be executed 'now' but it's effect on the environment is delayed by 'T' units of time.

The environment however is active in that time and there are other conditions based on which rewards are computed and returned. How is this handled?

I believe the Q value function (with gamma) handles 'delayed' effects of rewards but not actions.

This is similar to the inventory management use cases. As an analogy consider that I sell cakes. As customers walk into my shop I consume cakes off the shelf. I must reorder to stock my shelf BUT this reordering can take time to take effect.

I thought of just adding the quantity reordered to the shelf at a later time and let the agent learn it's effects. Will this suffice?

As another approach I thought of Experience and Replay as a mechanism to handle this delayed effect.

Appreciate the help.

I thought of Experience and Replay as a mechanism to handle this delayed effect.

Experience Replay will not help directly here, it is a mechanism to make learning more stable and effective from limited data, but it will not address delayed returns.

Any approach which learns Q should cope fine with delayed rewards. So Monte Carlo Control, SARSA, Q-learning, DQN and all their variants are in theory capable of learning the delayed reward. The total reward is the ultimate effect of any action.

However, I'm guessing just because it is not mentioned and you are asking this question, that you may still have a problem . . .

You claim that the action has a "delayed effect on the environment". You need to change that, because it implies that your state representation is incorrect for your problem. Clearly making an order for something changes the environment. An environment with an order in progress is different, in a way that is critically important to your problem, than one without an order in progress.

Probably what you are missing is a state representation that captures what your action has actually done. Without that, there are hidden variables (orders currently being handled) that a Q function cannot learn about because it is not in the state, $s$ for $Q(s,a)$. For RL to be reliable, the state value $s$ has to capture all relevant information about how future state changes and rewards will progress. For instance, if you were writing an agent to control a swinging pendulum (a standard toy problem), then you don't just need the position of the pendulum, you also want its velocity in order to predict where the pendulum might end up before any action is taken. For your issue, the current stock is like the pendulum's position, and you need to track orders "in progress" as they are a bit like velocity in that they will cause further changes to state regardless of action.

To address this, you will need to add a representation of recent stock orders to your state. I would guess something that describes the contents of the order (similar to your current stock representation) and a countdown for how many days left until it is delivered (or if they are not that reliable, something similar that correlates with likely arrival time). That state should be changed immediately in response to the action that caused the order, otherwise the agent will not learn the association between action and its effects, and will treat orders arriving as some kind of random effect from the environment.

Once you have a representation like this, then I think that is enough, and the next steps are finding the right hyper-parameter values to learn effectively. This will be easier initially if you have a simulated environment, although you could also learn off-policy from historical data using something like DQN, provided you have a lot of historical data to work with.

I thought of just adding the quantity reordered to the shelf at a later time and let the agent learn it's effects. Will this suffice?

I think this is similar, but probably not enough. The most important thing to do is associate change to state with the action that caused it. Adding the new stock as it arrives you should do anyway, but the agent needs to learn that the earlier action is what caused this, and that is only possible if that action actually changes the state in some way.

• @RajeshS: You need to re-frame that, as I state in this answer. The action is not delayed, it is taken when it is taken. The effects on the stock is delayed compared to the action. Your (implied) problem is that your state is currently just the stock and loses the information about orders in progress. This answer tells you how to fix that – Neil Slater Jul 19 '18 at 8:25
• @RajeshS: I have tried to make that clearer in the answer. – Neil Slater Jul 19 '18 at 8:34
• In your answer you say to improve the state representation and give the pendulum/velocity example - but in that case the impact of the next action taken is directly felt in the next state. The same can't be said for the example using a countdown. Isn't the issue here that the reward isn't properly reflecting the impact of the action rather than the state representation per se? If you can assign a reward at the very end of the episode instead of at every timestep doesn't this alleviate this issue? – tryingtolearn Feb 3 at 11:13
• @tryingtolearn: You are correct that there is a difference between generally delayed rewards and missing measured state. Which makes the pendulum example not 100% exact match to OP's problem. However, the issue here is still that the OP has not included relevant state. Changing the reward signal won't help unless it can somehow be assigned immediately (so that the delay is not noticed by the learning algorithm). If you want to explore/ask further about this, please ask a new question on the site. – Neil Slater Feb 3 at 11:21