Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Following this explanation on what is experience replay (and others), I noticed an experience element is defined as
$e_t = (s_t,a_t,r_t,s_{t+1})$
My question is, why do we need the next state in the experience?
To my understanding, our networks learn state to action and action to reward mappings, so I fail to see where the "next state" is used in experience replay?
True, but if you take a look at the equations of Q learning or Advantage functions in Policy Gradients you will see that the expected value of the next state is being used. For this, you need to know in which state you have landed from state $s_t$. For example, you can approximate your future reward from state $s_{t+1}$ by using $Q(s_{t+1})$. To do this you need to give to your network as input the $s_{t+1}$.
