0
$\begingroup$

Based on some papers which I read, distributed deep learning can provide faster training time. In addition, it also provides better accuracy or lower prediction error. What are the reasons?

Question edited:

I am using Tensorflow to run distributed deep learning (DL) and compare the performance with non-distributed DL. I use the number of dataset 1000 samples and step size 10000. The distributed DL uses 2 workers and 1 parameter server. Then, the following cases are considered when running the code:

  1. Each worker and non-distributed DL use 1000 samples for training sets, same mini-batch size 200

  2. Each worker uses 500 samples for training sets (first 500 samples for worker 1 and the rest 500 samples for worker 2), non-distributed DL use 1000 samples for training sets, same mini-batch size 200

  3. Each worker uses 500 samples for training sets (first 500 samples for worker 1 and the rest 500 samples for worker 2) with mini-batch size 100, non-distributed DL use 1000 samples for training sets with mini-batch size 200

Based on the simulation, for all cases, distributed DL has lower RMSE than non-distributed DL. In this case, the RMSEs of distributed DL are as follows: Distributed DL in Case 2 < Distributed DL in Case 1 < Distributed DL in Case 3 < Non-distributed.

In addition, I also add the training time (i.e., the number of steps is 2 x 10000) for non-distributed DL, the results are still not as good as distributed DL.

One reason can be the mini-batch size, however, I wonder the other reasons why the distributed DL has better performance using the aforementioned cases?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

About the accuracy: Going with the strongest reason; memory problems will diminish due to the distribution of the computation. That will allow you to increase your training batch size which will reduce the gradient noise due to small mini-batch sizes. The steeper gradient moves will be towards the minima, with less noise.

You can refer to this video for deeper understanding: https://www.youtube.com/watch?v=-_4Zi8fCZO4&list=PLkDaE6sCZn6Hn0vK8co82zjQtt3T2Nkqc&index=16

About the speed: It is more obvious I think. You distribute your gradient descent computations to multiple machines or CPUs/GPUs/TPUs, so a faster training speed you acquire as a result.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the explanation. Is there the only reason why the accuracy is better in distributed deep learning? Is there any influence coming from learning rate or the way we update the parameters as we need to accumulate gradients at each iteration? $\endgroup$
    – bnbfreak
    Nov 5, 2018 at 23:03
  • $\begingroup$ I think stating only one reason would be overdetermination, sorry for that. Mini-batch size claims to be the leading cause of performance; since increasing it makes your gradients steeper towards the minima. Note: when you change the batch size, you will need to tune the others eventually, especially the learning rate. (if it is too large, you can miss the minima, due to the larger steps that gradient update will take due to the batch-size). The way that gradient is updated would not change unless you work with a framework that allows you to use different configurations for each worker. $\endgroup$
    – Ugur MULUK
    Nov 6, 2018 at 11:13
  • $\begingroup$ Hi. I have updated the questions with my detail experiment. How do you think? Is that also because the mini-batch size influence? $\endgroup$
    – bnbfreak
    Nov 6, 2018 at 23:10
  • $\begingroup$ As i explained, you need to tune the others when you change the mini-batch size. It is possible for your gradient algorithm to miss the minima due to the larger steps it take because of the larger batch sizes. On the other hand, number of samples you have are not sufficient for experimenting such performance differences, those amounts will lead only to overfitting; I think RMSEs you acquire are random. If I had 500 samples I would go for directly batch gradient descent where you do not have any computational efficiency problem. You need at least 10,000+ samples to work on this by rule of hand. $\endgroup$
    – Ugur MULUK
    Nov 7, 2018 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.