# RL: Collecting States (training data) in real-life. Must use fixed timestep?

I am using a Reinforcement Learning agent to play a 3D game, but have trouble with collecting the "current and next state" pairs.

To decide what action to perform, the network must perform a forward pass.

It performs forward pass in time $$t$$, but in the meantime the actual environment might have evolved significantly. In computer games this means something like 10 frames or more have already passed (a varying amount).

The situation is worsened if I run, say 100 games at once on the same computer.

I don't have the ability to stop the game at each frame to do forwardprop. Anyway it wouldn't be possible were I to train, say a real-life robot to walk.

Question:

Should I stick to a 'fixed timestep' approach, only asking to provide an action every 0.1 seconds? While it computes next action, I could pretend the network keeps outputting the most recent action for all the skipped frames. Good idea?

If that's the only option, then should I avoid at all costs situations where forward prop takes more than the 'fixed timestep'? (more than 0.1 sec in my case) So it's better to choose say, 0.2 seconds just to be safe.

Seems quite unreliable - is there a better way to do it?

Is there a paper that explores the alternatives? (I guess such a paper will be about real-life robot training)

Your "fixed timestep" idea is actually very similar to a common technique called frame skipping. Instead of waiting a fixed amount of time, agents wait a fixed number of frames $$k$$ before choosing a new action. In the meantime, they repeat their most recently chosen action.

Frame skipping was included as part of the Atari 2600 Arcade Learning Environment. It was also used in the foundational DQN paper. Common values of $$k$$ are 3, 4, and 5. The value chosen depended on the game being played, since different games had important events happen at different time resolutions. In these papers, frame skipping enabled training to happen roughly $$k$$ times faster. So this is definitely a valid technique to try.

I actually think this would generally be less of a concern in the robotics application. Forward propagation usually happens much more quickly than real-world event timescales. As an example, Stanford famously applies RL to fly small helicopters, which requires considerable precision.

Finally, if your forward propagation really is taking too long, you should consider a faster architecture. One approach would be just to make your neural net smaller. You might consider policy distillation for compressing a large, trained network into a smaller one. Also, make sure you're not using some ridiculously slow activation function like sigmoid or tanh. ReLU is the common choice if you don't need a bounded output for a given neuron. If you do, I recommend softsign.

If your time bottleneck is actually in action selection, due to a large action space and using a value network, you should seriously consider switching to a policy-based method (e.g. actor critic). This would help because sampling from a distribution over actions would potentially be much faster than the $$\max$$ operation involved in value-based methods. You can read more about this in Section 13.7 of Sutton and Barto's RL book.

"Experience replay" lets agents remember and reuse experiences from the past. Experience replay bundles previous experiences (state and action pairs) which allows training on sparse rewards. Experience replay stores the agent's experiences in a buffer. Then train the agent against the entire contents of the buffer. If state and action pairs in the buffer yield a reward, they are reinforced. If state and action pairs in the buffer did not yield a reward, they are not reinforced.

The training signal at the end of the buffer is called "experience transitions". It is typically uniformly sampled. However, important experiences can be also prioritized.

"A Deeper Look at Experience Replay" paper goes into greater detail about the technique.

• Thank you Brian, but my question is not necessarily about experience replay. I am interested in collecting the actual data such that it doesn't go out of sync with the environment that is dynamically changing (while we are collecting data). – Kari Mar 21 '19 at 4:24
• Experience replay is an algorithm that does not care if the data is not in sync for the environment. It is robust to changes in the environment, thus solves the problem without needing to change data collection methods. – Brian Spiering Mar 21 '19 at 19:18