Something that confuses me with machine learning in practice is that when you send in a batch of examples, you are computing the value of each example independently.

And then when you backprop, you are computing the gradient also independently.

That seems to be a great deal of wasted resources. But obvious in practice, when training a network on Pytorch or Tensorflow, this is not done sequentially, and it scales well even if you change your batch to a relatively high number in the hundreds. How is this process sped up in practice?

I don't know too much about "parallel computation" so I am not confident to say that this is some magical parallel computing. I know that computation can be done in parallel on multi-thread CPUs, multi-core CPUs, hyperthreading, GPUs and FPGAs, or some non-trivial good old linear alg. Which one is actually used in practice?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.