I read a paper on Deep Neural Networks Compression (link: https://openreview.net/forum?id=SkhQHMW0W) and came across a term "gradient exchange", I tried making sense out of it but couldn't exactly understand what it intuitively refers to. What does the term gradient exchange mean?
Gradient exchange occurs in distributed learning systems that perform gradient descent, when one part of the distributed system need to use the gradient values from another part in order to complete a task.
For example, you may distribute a large data set between multiple nodes, and want to calculate a gradient descent step as part of optimisation. One way to do so is calculate a subset of batch gradients on each node and collate them at a single node in order to alter parameters synchronously. This means it is necessary to fetch gradients from all nodes into a single node so that a combined gradient for some weight parameters can be calculated and the parameters updated consistently in the update step.
Gradient exchange is just a term to describe that event - node A needs some gradients that node B has calculated, so they are requested (or pushed) and have to travel between the nodes. This is a relatively slow I/O process - it is necessary for distributed system to work, but for high performance you want to minimise time spent moving the data.
Other data (such as the parameters) also needs to be shared between nodes. This particular piece of data regarding the gradients is singled out for the paper as the authors have discovered a way to compress it significantly without losing performance of the learning algorithm. This is partly because gradients can be treated approximately in the first place. Many learning algorithms further adjust or normalise gradients after they have been calculated, so using super-precise values is not as important as you might think.
There may also be clever ways of splitting the update work so that each node only needs some of the gradients and only updates some of the parameters at each step. That will keep node CPU busy, possibly at the expense of more complicated communications. I do not know the details of any optimised distributed learning system in order to tell you the precise data exchanges and optimisations taking place. There are likely to be a few variations possible, depending on the framework and which algorithm is being implemented.