My question may sound like a duplicate of, for example, How is that possible that a reward function depends both on the next state and an action from current state? but I still feel confused.
In neural network approximation of the Q function, I follow the experience replay routine. Papers and manuals suggest storing state, action, received reward, and next state information in the experience replay. However when one is about to calculate a maximum of Q value based on the next action, which state do they use: state or next state? If the next state is used to calculate the max Q, then what is the whole purpose of storing the previous state and action information?
An example (2D world):
An agent is at cell A1 (state). His goal is to get to C3 to get a positive reward. He then moves to B2 (action); a received reward is, let's say, 0. Next state is B2 (after the action was done).
Should one rely on the B2 state to iterate over possible actions from this state (next state) to get an approximation of highest reward (max Q)?
Then, why do we store the A1 and move-to-B2 information at all in the replay buffer?
Or I am wrong and we just use the A1 and iterate over possible actions (including that to B2) to get the max Q?
Edit: I think I have found an answer ). We need to store previous state (A1) and action (move to B2) in order to create the state-action distribution, which will be met with the expected long-term reward distribution, that we get after the next state routine. Right?