2
$\begingroup$

I would like to know if there is a way to train a deep rl model using two different computers. The first one would execute the game and send requisitions to the second computer which would store and train the model itself.

Obs: The computers aren't in the same LAN.

Thanks!!

$\endgroup$
1
$\begingroup$

Distributed RL is very much a thing. Google have created a distributed setup called IMPALA for this, and there are multiple instances of A3C, PPO etc available if you search. I don't know much about IMPALA, but the basic idea of the scalable policy gradient methods is to run multiple environments, collecting gradients on each server, then collating them together to create improved policy and value networks every few steps.

There are a couple of variations in strategy based on which stage of the data is shared - observations or gradients. Gradient calculation is CPU intensive, so above a certain scale it is worth having that occur on the distributed devices, depending on how intensive it is to collect experience in the first place.

Obs: The computers aren't in the same LAN.

This may prevent you implementing anything with low-level observation or gradient sharing, unless the bandwidth between the machines is high.

The simplest way to use two computers in this case is to perform basic hyper-parameter searches by running different tests on each computer and tracking which computer has done which experiments.

The first one would execute the game and send requisitions to the second computer which would store and train the model itself

This could work with an off-policy value-based method, such as DQN. You still need a reasonable bandwidth between the two machines, especially if the observation space is large. DQN is a reasonable choice since you don't need the environment-running machine to follow the current optimal policy - although you will still want to update the policy on the first computer some of the time.

The basic algorithms of DQN do not require much changing to support this kind of distribution. Just comment out or make conditional a few parts:

  • On the first machine:

    • Comment out or logically block sampling and learning from experience table
    • Maintain a "behaviour-driving" Q-value network instead of the learning network, in order to run $\epsilon$-greedy policy
    • Send experience to second machine instead of storing it in local experience-replay table (this is the bandwidth-intensive part)
    • Asynchronously receive updates to behaviour-driving Q-value network
  • On the second machine:

    • Comment out or logically block interactions with the environment
    • Asynchronously receive experience from first machine and add to experience-replay table
    • Every so many mini-batches, send updated current network to second machine

You will need to have some experience handling multi-threading or multi-process code in order to cover the asynchronous nature of the updates. If that seems too hard, then you could have the two machines attempt to update each other synchronously, or via a queueing mechanism. The disadvantage of that is one or other machine could end up idle waiting for its partner to complete its part of the work.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Hello, Neil. Thank you for the comprehensive and enlightening answer!! I'm currently studying the dqn technique and what you said makes a lot of sense. I'll definetely go deep in this Impala paper and try to experiment a little things on this. Have you tried something like that? Thanks so much again!! $\endgroup$ – Matheus Prandini Nov 8 '18 at 13:22
  • 1
    $\begingroup$ @MatheusPrandini: No I have not tried any multiple-machine setup, I have only read some of the theory, but it was very recently that I was reading about it, so it was fresh in my mind. $\endgroup$ – Neil Slater Nov 8 '18 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.