Suppose I want to develop and train a big end-to-end deep learning model using Tensorflow (1.15, for legacy reasons). The objects are complex, with many types of features that can be extracted: vector of numeric features of fixed length, sequences, unordered sets, etc. Thus, the model will include many submodules to deal with various types of features.
I have access to a server with several GPUs, so I want to distribute the model across them. What is the best way to do so? So far I'm thinking about placing subsystems on separate GPUs, but this presents some questions:
- How costly would be the transfer of computation results between GPUs? Tensorflow does it automatically, right?
- How costly would gradient computation and descent be, considering variables are placed on different GPUs? Would gradients also be computed on the same GPUs as their corresponding variables?