I can see two motives to use Synthetic Gradients in RNN:

1. To speed up training, by imediately correcting each layer with predicted gradient
2. To be able to learn longer sequences

I see problems with both of them. Please note, I really like Synthetic Gradients and would like to implement them. But I need to understand where my trail of thought is incorrect.

I will now show why Point 1 and Point 2 don't seem to be beneficial, and I need you to correct me, if they are actually beneficial:

Point 1:

Synthetic Gradients tell us we can rely on another "mini-helper-network" (called DNI) to advise our current layer about what gradients will arrive from above, even during fwd prop.

However, such gradients will only come several operations later. Same amount of Backprop will have to be done as without DNI, except that now we also need to train our DNI.

Adding this Asyncronisity shouldn't not make layers train faster than during the traditional "locked" full fwdprop -> full back prop sequence, because same number of computations must be done by the device. It's just that the computations will be slid in time

This makes me think Point 1) will not work. Simply adding SG between each layer shouldn't improve training speed.

Point 2:

Ok, how about adding SG only on the last layer to predict "gradient from future" and only if it's the final timestep during forward prop.

This way, even though our LSTM has to stop predicting and must backpropagate, it can still predict the future-gradient it would have received (with the help of DNI sitting on the last timestep).

Consider several training sessions (session A, session B):

fwdprop timestep_1A ---> fwdprop timestep_2A ---> fwdprop timestep_3A ----> stop and bkprop!

fwdprop timestep_1B ---> fwdprop timestep_2B ---> fwdprop timestep_3B ----> stop and bkprop!

 

We've just forced our network to "parse" 6 timesteps in two halves: 3 timesteps, then 3 remaining timesteps again.

Notice, we have our DNI sitting at the very end of "Session A" and predicting "what gradient I would get flowing from the beginning of Session B (from future)". Because of that, timestep_3A will be equipped with gradient "that would have come from timestep_1B", so indeed, corrections done during A will be more reliable.

But, hey! These predicted "synthetic gradients" will be very small (negligible) anyway - after all, that's why we start a new backprop session B. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop "session A".

Therefore I think Point 2) shouldn't give benefit either. Adding SG on the last timestep of fwdprop allows to effectively train longer sequences, but vanishing gradients didn't go anywhere.

Ok. Maybe we can get the benefit of training "session A", "session B" etc on separate machines? But then how is this different to simply training with the usual minibatches in parallel? Keep in mind, was mentioned in point 2: things are worsened by sessionA predicting gradients which are vanishing anyway.

Question: Please help me understand the benefit of Synthetic Gradient, because the 2 points above don't seem to be beneficial

I can see two motives to use Synthetic Gradients in RNN:

1. To speed up training, by imediately correcting each layer with predicted gradient
2. To be able to learn longer sequences

I see problems with both of them. Please note, I really like Synthetic Gradients and would like to implement them. But I need to understand where my trail of thought is incorrect.

I will now show why Point 1 and Point 2 don't seem to be beneficial, and I need you to correct me, if they are actually beneficial:

Point 1:

Synthetic Gradients tell us we can rely on another "mini-helper-network" (called DNI) to advise our current layer about what gradients will arrive from above, even during fwd prop.

However, such gradients will only come several operations later. Same amount of Backprop will have to be done as without DNI, except that now we also need to train our DNI.

Adding this Asyncronisity shouldn't not make layers train faster than during the traditional "locked" full fwdprop -> full back prop sequence, because same number of computations must be done by the device. It's just that the computations will be slid in time

This makes me think Point 1) will not work. Simply adding SG between each layer shouldn't improve training speed.

Point 2:

Ok, how about adding SG only on the last layer to predict "gradient from future" and only if it's the final timestep during forward prop.

This way, even though our LSTM has to stop predicting and must backpropagate, it can still predict the future-gradient it would have received (with the help of DNI sitting on the last timestep).

Consider several training sessions (session A, session B):

fwdprop timestep_1A ---> fwdprop timestep_2A ---> fwdprop timestep_3A ----> stop and bkprop!

fwdprop timestep_1B ---> fwdprop timestep_2B ---> fwdprop timestep_3B ----> stop and bkprop!

We've just forced our network to "parse" 6 timesteps in two halves: 3 timesteps, then 3 remaining timesteps again.

Notice, we have our DNI sitting at the very end of "Session A" and predicting "what gradient I would get flowing from the beginning of Session B (from future)". Because of that, timestep_3A will be equipped with gradient "that would have come from timestep_1B", so indeed, corrections done during A will be more reliable.

But, hey! These predicted "synthetic gradients" will be very small (negligible) anyway - after all, that's why we start a new backprop session B. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop "session A".

Therefore I think Point 2) shouldn't give benefit either. Adding SG on the last timestep of fwdprop allows to effectively train longer sequences, but vanishing gradients didn't go anywhere.

Ok. Maybe we can get the benefit of training "session A", "session B" etc on separate machines? But then how is this different to simply training with the usual minibatches in parallel? Keep in mind, was mentioned in point 2: things are worsened by sessionA predicting gradients which are vanishing anyway.

Question: Please help me understand the benefit of Synthetic Gradient, because the 2 points above don't seem to be beneficial

I can see two motives to use Synthetic Gradients in RNN:

1. To speed up training, by imediately correcting each layer with predicted gradient
2. To be able to learn longer sequences

I see problems with both of them. Please note, I really like Synthetic Gradients and would like to implement them. But I need to understand where my trail of thought is incorrect.

I will now show why Point 1 and Point 2 don't seem to be beneficial, and I need you to correct me, if they are actually beneficial:

Synthetic Gradients tell us we can rely on another "mini-helper-network" (called DNI) to advise our current layer about what gradients will arrive from above, even during fwd prop.

However, such gradients will only come several operations later. Same amount of Backprop will have to be done as without DNI, except that now we also need to train our DNI.

Adding this Asyncronisity shouldn't not make layers train faster than during the traditional "locked" full fwdprop -> full back prop sequence, because same number of computations must be done by the device. It's just that the computations will be slid in time

This makes me think Point 1) will not work. Simply adding SG between each layer shouldn't improve training speed.

Ok, how about adding SG only on the last layer to predict "gradient from future" and only if it's the final timestep during forward prop.

This way, even though our LSTM has to stop predicting and must backpropagate, it can still predict the future-gradient it would have received (with the help of DNI sitting on the last timestep).

Consider several training sessions (session A, session B):

fwdprop timestep_1A ---> fwdprop timestep_2A ---> fwdprop timestep_3A ----> stop and bkprop!

fwdprop timestep_1B ---> fwdprop timestep_2B ---> fwdprop timestep_3B ----> stop and bkprop!

We've just forced our network to "parse" 6 timesteps in two halves: 3 timesteps, then 3 remaining timesteps again.

Notice, we have our DNI sitting at the very end of "Session A" and predicting "what gradient I would get flowing from the beginning of Session B (from future)". Because of that, timestep_3A will be equipped with gradient "that would have come from timestep_1B", so indeed, corrections done during A will be more reliable.

But, hey! These predicted "synthetic gradients" will be very small (negligible) anyway - after all, that's why we start a new backprop session B. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop "session A".

Therefore I think Point 2) shouldn't give benefit either. Adding SG on the last timestep of fwdprop allows to effectively train longer sequences, but vanishing gradients didn't go anywhere.

Ok. Maybe we can get the benefit of training "session A", "session B" etc on separate machines? But then how is this different to simply training with the usual minibatches in parallel? Keep in mind, was mentioned in point 2: things are worsened by sessionA predicting gradients which are vanishing anyway.

Question: Please help me understand the benefit of Synthetic Gradient, because the 2 points above don't seem to be beneficial

I can see two motives to use Synthetic Gradients in RNN:

1. To speed up training, by imediately correcting each layer with predicted gradient
2. To be able to learn longer sequences

I see problems with both of them. Please note, I really like Synthetic Gradients and would like to implement them. But I need to understand where my trail of thought is incorrect.

I will now show why Point 1 and Point 2 don't seem to be beneficial, and I need you to correct me, if they are actually beneficial:

Point 1:

Synthetic Gradients tell us we can rely on another "mini-helper-network" (called DNI) to advise our current layer about what gradients will arrive from above, even during fwd prop.

However, such gradients will only come several operations later. Same amount of Backprop will have to be done as without DNI, except that now we also need to train our DNI.

Adding this Asyncronisity shouldn't not make layers train faster than during the traditional "locked" full fwdprop -> full back prop sequence, because same number of computations must be done by the device. It's just that the computations will be slid in time

This makes me think Point 1) will not work. Simply adding SG between each layer shouldn't improve training speed.

Point 2:

Ok, how about adding SG only on the last layer to predict "gradient from future" and only if it's the final timestep during forward prop.

This way, even though our LSTM has to stop predicting and must backpropagate, it can still predict the future-gradient it would have received (with the help of DNI sitting on the last timestep).

Consider several training sessions (session A, session B):

fwdprop timestep_1A ---> fwdprop timestep_2A ---> fwdprop timestep_3A ----> stop and bkprop!

fwdprop timestep_1B ---> fwdprop timestep_2B ---> fwdprop timestep_3B ----> stop and bkprop!

We've just forced our network to "parse" 6 timesteps in two halves: 3 timesteps, then 3 remaining timesteps again.

Notice, we have our DNI sitting at the very end of "Session A" and predicting "what gradient I would get flowing from the beginning of Session B (from future)". Because of that, timestep_3A will be equipped with gradient "that would have come from timestep_1B", so indeed, corrections done during A will be more reliable.

But, hey! These predicted "synthetic gradients" will be very small (negligible) anyway - after all, that's why we start a new backprop session B. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop "session A".

Therefore I think Point 2) shouldn't give benefit either. Adding SG on the last timestep of fwdprop allows to effectively train longer sequences, but vanishing gradients didn't go anywhere.

Ok. Maybe we can get the benefit of training "session A", "session B" etc on separate machines? But then how is this different to simply training with the usual minibatches in parallel? Keep in mind, was mentioned in point 2: things are worsened by sessionA predicting gradients which are vanishing anyway.

Question: Please help me understand the benefit of Synthetic Gradient, because the 2 points above don't seem to be beneficial

I can see two motives to use Synthetic Gradients in RNN:

1. To speed up training, by imediately correcting each layer with predicted gradient
2. To be able to learn longer sequences

I see problems with both of them. Please note, I really like Synthetic Gradients and would like to implement them. But I need to understand where my trail of thought is incorrect.

I will now show why Point 1 and Point 2 don't seem to be beneficial, and I need you to correct me, if it actualy is beneficial:

Synthetic Gradients tell us we can rely on another "mini-helper-network" (called DNI) to advise our current layer about what gradients will arrive from above, even during fwd prop.

However, such gradients will only come several operations later. Same amount of Backprop will have to be done as without DNI, except that now we also need to train our DNI.

Adding this Asyncronisity shouldn't not make layers train faster than during the traditional "locked" full fwdprop -> full back prop sequence, because same number of computations must be done by the device. It's just that the computations will be slid in time

This makes me think Point 1) will not work. Simply adding SG between each layer shouldn't improve training speed.

Ok, how about adding SG only on the last layer to predict "gradient from future" and only if it's the final timestep during forward prop.

This way, even though our LSTM has to stop predicting and must backpropagate, it can still predict the future-gradient it would have received (with the help of DNI sitting on the last timestep).

Consider several training sessions (session A, session B):

fwdprop timestep_1A ---> fwdprop timestep_2A ---> fwdprop timestep_3A ----> stop and bkprop!

fwdprop timestep_1B ---> fwdprop timestep_2B ---> fwdprop timestep_3B ----> stop and bkprop!

We've just forced our network to "parse" 6 timesteps in two halves: 3 timesteps, then 3 remaining timesteps again.

Notice, we have our DNI sitting at the very end of "Session A" and predicting "what gradient I would get flowing from the beginning of Session B (from future)". Because of that, timestep_3A will be equipped with gradient "that would have come from timestep_1B", so indeed, corrections done during A will be more reliable.

But, hey! These predicted "synthetic gradients" will be very small (negligible) anyway - after all, that's why we start a new backprop session B. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop "session A".

Therefore I think Point 2) shouldn't give benefit either. Adding SG on the last timestep of fwdprop allows to effectively train longer sequences, but vanishing gradients didn't go anywhere.

Ok. Maybe we can get the benefit of training "session A", "session B" etc on separate machines? But then how is this different to simply training with the usual minibatches in parallel? Keep in mind, was mentioned in point 2: things are worsened by sessionA predicting gradients which are vanishing anyway.

Question: Please help me understand the benefit of Synthetic Gradient, because the 2 points above don't seem to be beneficial

I can see two motives to use Synthetic Gradients in RNN:

1. To speed up training, by imediately correcting each layer with predicted gradient
2. To be able to learn longer sequences

I see problems with both of them. Please note, I really like Synthetic Gradients and would like to implement them. But I need to understand where my trail of thought is incorrect.

I will now show why Point 1 and Point 2 don't seem to be beneficial, and I need you to correct me, if they are actually beneficial:

Synthetic Gradients tell us we can rely on another "mini-helper-network" (called DNI) to advise our current layer about what gradients will arrive from above, even during fwd prop.

However, such gradients will only come several operations later. Same amount of Backprop will have to be done as without DNI, except that now we also need to train our DNI.

Adding this Asyncronisity shouldn't not make layers train faster than during the traditional "locked" full fwdprop -> full back prop sequence, because same number of computations must be done by the device. It's just that the computations will be slid in time

This makes me think Point 1) will not work. Simply adding SG between each layer shouldn't improve training speed.

Ok, how about adding SG only on the last layer to predict "gradient from future" and only if it's the final timestep during forward prop.

This way, even though our LSTM has to stop predicting and must backpropagate, it can still predict the future-gradient it would have received (with the help of DNI sitting on the last timestep).

Consider several training sessions (session A, session B):

fwdprop timestep_1A ---> fwdprop timestep_2A ---> fwdprop timestep_3A ----> stop and bkprop!

fwdprop timestep_1B ---> fwdprop timestep_2B ---> fwdprop timestep_3B ----> stop and bkprop!

We've just forced our network to "parse" 6 timesteps in two halves: 3 timesteps, then 3 remaining timesteps again.

Notice, we have our DNI sitting at the very end of "Session A" and predicting "what gradient I would get flowing from the beginning of Session B (from future)". Because of that, timestep_3A will be equipped with gradient "that would have come from timestep_1B", so indeed, corrections done during A will be more reliable.

But, hey! These predicted "synthetic gradients" will be very small (negligible) anyway - after all, that's why we start a new backprop session B. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop "session A".

Therefore I think Point 2) shouldn't give benefit either. Adding SG on the last timestep of fwdprop allows to effectively train longer sequences, but vanishing gradients didn't go anywhere.

Ok. Maybe we can get the benefit of training "session A", "session B" etc on separate machines? But then how is this different to simply training with the usual minibatches in parallel? Keep in mind, was mentioned in point 2: things are worsened by sessionA predicting gradients which are vanishing anyway.

Question: Please help me understand the benefit of Synthetic Gradient, because the 2 points above don't seem to be beneficial