Train a deep reinforcement learning model using two computers

Question

I would like to know if there is a way to train a deep rl model using two different computers. The first one would execute the game and send requisitions to the second computer which would store and train the model itself.

Obs: The computers aren't in the same LAN.

Thanks!!

Answer 1

Distributed RL is very much a thing. Google have created a distributed setup called IMPALA for this, and there are multiple instances of A3C, PPO etc available if you search. I don't know much about IMPALA, but the basic idea of the scalable policy gradient methods is to run multiple environments, collecting gradients on each server, then collating them together to create improved policy and value networks every few steps.

There are a couple of variations in strategy based on which stage of the data is shared - observations or gradients. Gradient calculation is CPU intensive, so above a certain scale it is worth having that occur on the distributed devices, depending on how intensive it is to collect experience in the first place.

Obs: The computers aren't in the same LAN.

This may prevent you implementing anything with low-level observation or gradient sharing, unless the bandwidth between the machines is high.

The simplest way to use two computers in this case is to perform basic hyper-parameter searches by running different tests on each computer and tracking which computer has done which experiments.

The first one would execute the game and send requisitions to the second computer which would store and train the model itself

This could work with an off-policy value-based method, such as DQN. You still need a reasonable bandwidth between the two machines, especially if the observation space is large. DQN is a reasonable choice since you don't need the environment-running machine to follow the current optimal policy - although you will still want to update the policy on the first computer some of the time.

The basic algorithms of DQN do not require much changing to support this kind of distribution. Just comment out or make conditional a few parts:

• On the first machine:

  • Comment out or logically block sampling and learning from experience table
  • Maintain a "behaviour-driving" Q-value network instead of the learning network, in order to run $\epsilon$-greedy policy
  • Send experience to second machine instead of storing it in local experience-replay table (this is the bandwidth-intensive part)
  • Asynchronously receive updates to behaviour-driving Q-value network

• On the second machine:

  • Comment out or logically block interactions with the environment
  • Asynchronously receive experience from first machine and add to experience-replay table
  • Every so many mini-batches, send updated current network to second machine

You will need to have some experience handling multi-threading or multi-process code in order to cover the asynchronous nature of the updates. If that seems too hard, then you could have the two machines attempt to update each other synchronously, or via a queueing mechanism. The disadvantage of that is one or other machine could end up idle waiting for its partner to complete its part of the work.