# How does generalised advantage estimation work?

I've been trying to add GAE to my A2C implementation for a while now, but I' can't quite seem to grok how it works.

My understanding of it, is that it reduces the variance of the advantage estimation function by kind of 'averaging out' (or generalising) the advantages based off the values in the rollout.

I tried to run through the maths on my own, and in the end I just had one advantage for the whole rollout, is this right? Normally, we'd have one advantage for each timestep in the rollout.

Can anyone provide an explanation on the intuition of GAE?

## 2 Answers

I found very intuitive the explanation of the GAE in the Supplementary material of this paper: DeepMimic. You do not need to read the paper. Just go straight to the Supplementary material section on page 143:15. For the λ-return you can find lots of information in the Reinforcement Learning book of Sutton and Barto. Hope it helps!

• Good catch, that's the clearest explanation of λ-return I've read so far! – MasterScrat Feb 25 '19 at 13:20

You can find a good explanation of λ-return on Lilian Weng's blog: Combining TD and MC Learning .

The Generalized Advantage Estimator GAE(λ) simply uses λ-return to estimate the advantage function.