Something that confuses me with machine learning in practice is that when you send in a batch of examples, you are computing the value of each example independently.
And then when you backprop, you are computing the gradient also independently.
That seems to be a great deal of wasted resources. But obvious in practice, when training a network on Pytorch or Tensorflow, this is not done sequentially, and it scales well even if you change your batch to a relatively high number in the hundreds. How is this process sped up in practice?
I don't know too much about "parallel computation" so I am not confident to say that this is some magical parallel computing. I know that computation can be done in parallel on multi-thread CPUs, multi-core CPUs, hyperthreading, GPUs and FPGAs, or some non-trivial good old linear alg. Which one is actually used in practice?