0
$\begingroup$

Something that confuses me with machine learning in practice is that when you send in a batch of examples, you are computing the value of each example independently.

And then when you backprop, you are computing the gradient also independently.

That seems to be a great deal of wasted resources. But obvious in practice, when training a network on Pytorch or Tensorflow, this is not done sequentially, and it scales well even if you change your batch to a relatively high number in the hundreds. How is this process sped up in practice?

I don't know too much about "parallel computation" so I am not confident to say that this is some magical parallel computing. I know that computation can be done in parallel on multi-thread CPUs, multi-core CPUs, hyperthreading, GPUs and FPGAs, or some non-trivial good old linear alg. Which one is actually used in practice?

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.