What is the difference between Pytorch's DataParallel and DistributedDataParallel?

I am going through this imagenet example.

And, in line 88, the module DistributedDataParallel is used. When I searched for the same in the docs, I haven’t found anything. However, I found the documentation for DataParallel.

So, would like to know what is the difference between the DataParallel and DistributedDataParallel modules.

• well, by docs I think they are totally same. Aren't they? – Frank Jan 10 at 5:47

As the Distributed GPUs functionality is only a couple of days old [in the v2.0 release version of Pytorch], there is still no documentation regarding that. So, I had to go through the source code's docstrings for figuring out the difference. So, the docstring of the DistributedDataParallel module is as follows:
And the docstring for the dataparallel is as follows: