6
$\begingroup$

I've been trying to add GAE to my A2C implementation for a while now, but I' can't quite seem to grok how it works.

My understanding of it, is that it reduces the variance of the advantage estimation function by kind of 'averaging out' (or generalising) the advantages based off the values in the rollout.

I tried to run through the maths on my own, and in the end I just had one advantage for the whole rollout, is this right? Normally, we'd have one advantage for each timestep in the rollout.

Can anyone provide an explanation on the intuition of GAE?

|improve this question
$\endgroup$
7
$\begingroup$

I found very intuitive the explanation of the GAE in the Supplementary material of this paper: DeepMimic. You do not need to read the paper. Just go straight to the Supplementary material section on page 143:15. For the λ-return you can find lots of information in the Reinforcement Learning book of Sutton and Barto. Hope it helps!

|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Good catch, that's the clearest explanation of λ-return I've read so far! $\endgroup$ – MasterScrat Feb 25 at 13:20
1
$\begingroup$

You can find a good explanation of λ-return on Lilian Weng's blog: Combining TD and MC Learning .

The Generalized Advantage Estimator GAE(λ) simply uses λ-return to estimate the advantage function.

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.