Following this explanation on what is experience replay (and others), I noticed an experience element is defined as

$e_t = (s_t,a_t,r_t,s_{t+1})$

My question is, why do we need the next state in the experience?

To my understanding, our networks learn state to action and action to reward mappings, so I fail to see where the "next state" is used in experience replay?


1 Answer 1


True, but if you take a look at the equations of Q learning or Advantage functions in Policy Gradients you will see that the expected value of the next state is being used. For this, you need to know in which state you have landed from state $s_t$. For example, you can approximate your future reward from state $s_{t+1}$ by using $Q(s_{t+1})$. To do this you need to give to your network as input the $s_{t+1}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.