I want to use Prioritized Experience Replay for whole episodes, instead for single transitions.

What's the best way to define the priorities - as episodes can be of different lengths? Personally I can think of summing up TD(0) / TD(n) errors or averaging them. What would this mean for hyperparameters like α and β?

Is there actually a 'correct way' from theoretical perspective?

  • $\begingroup$ What return estimate are you using? When I saw the title I thought it would be Monte Carlo (or TD(1)) - is it single step TD (or TD(0)) or some intermediate step size (e.g. TD($\lambda$))? This could make a difference theoretically - although I am not sure whether there is a strong theoretical backing even to normal prioritized experience replay, it's more of an engineering approach backed by good empirical results. $\endgroup$ – Neil Slater Sep 19 '19 at 7:00
  • $\begingroup$ Currently I'm using TD(0), but I'm planning to extend it to n-step TD / TD(n) (but not TD(λ)). The reason why I want to store whole episodes is the recurrent nature of my data (and my model), so the model needs to replay whole episodes. $\endgroup$ – ScientiaEtVeritas Sep 19 '19 at 9:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.