0
$\begingroup$

I have access to a hpc node, of 3 GPU and maximum of 38 CPU. I have a transformer model which I run of a single GPU at the moment, I want to utilize all the GPUs and CPUs. I have seen couple of tutorial on Dataparrallel and DistributedDataParallel. They only mentioned how to use multiple GPUs.

My questions are:

  1. Do I use Dataparallel or DistributedDataParallel
  2. How do I adapt my code run on the GPUs and CPUs simultaneously. Perhaps if I can get a tutorial link.
  3. How to do I get the device ids.
Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

  1. I did use DistributedDataParallel according to PyTorch documentation DataParallel is usually slower than DistributedDataParallel therefore it is recommended since DistributedDataParallel works for both single- and multi-machine training.

  2. Tutorial Comparison between DataParallel and DistributedDataParallel

  3. Another Tutorial Multi-GPU Examples

  4. Solution LITDataScience's answer - How to find the nvidia GPU IDs for pytorch cuda run setup?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.