state-action-reward-new state: confusion of terms

Question

My question may sound like a duplicate of, for example, How is that possible that a reward function depends both on the next state and an action from current state? but I still feel confused.

In neural network approximation of the Q function, I follow the experience replay routine. Papers and manuals suggest storing state, action, received reward, and next state information in the experience replay. However when one is about to calculate a maximum of Q value based on the next action, which state do they use: state or next state? If the next state is used to calculate the max Q, then what is the whole purpose of storing the previous state and action information?

An example (2D world):

An agent is at cell A1 (state). His goal is to get to C3 to get a positive reward. He then moves to B2 (action); a received reward is, let's say, 0. Next state is B2 (after the action was done).

Questions:

Should one rely on the B2 state to iterate over possible actions from this state (next state) to get an approximation of highest reward (max Q)?

Then, why do we store the A1 and move-to-B2 information at all in the replay buffer?

Or I am wrong and we just use the A1 and iterate over possible actions (including that to B2) to get the max Q?

Edit: I think I have found an answer ). We need to store previous state (A1) and action (move to B2) in order to create the state-action distribution, which will be met with the expected long-term reward distribution, that we get after the next state routine. Right?

Answer 1

The TD Target (for learning update) for using $\hat{q}(s,a)$ neural network in Q-learning is:

$$r + \text{max}_{a'} \hat{q}(s',a')$$

In order to calculate this, you need a starting state $s$, the action taken form that state $a$, and the resulting reward $r$ and state $s'$.

You need the $s, a$ to generate the input to the neural network (what you might call train_X for supervised learning). You need $r, s'$ to generate the TD Target shown above as an output to learn for regression (in supervised learning, that would be train_y). And you need to work through all possible $a'$ based on $s'$ in order to find the maximum value for the TD Target equation used in Q learning - other RL algorithms may use variations of this for calculating the TD Target.

This means your suggestions are all close but not quite right.

Should one rely on the B2 state to iterate over possible actions from this state (next state) to get an approximation of highest reward (max Q)?

Sort of. The B2 state is $s'$ from the equation, so is responsible for calculating the TD target for state-action value $q(s,a)$.

Then, why do we store the A1 and move-to-B2 information at all in the replay buffer?

You still need to know $s$ is A1, because the representation of A1 (and whichever action is taken) will be the input to your network.

Or I am wrong and we just use the A1 and iterate over possible actions (including that to B2) to get the max Q?

Iterate over actions from B2 using your neural network to pick the highest estimate.

I think I have found an answer ). We need to store previous state (A1) and action (move to B2) in order to create the state-action distribution, which will be met with the expected long-term reward distribution, that we get after the next state routine. Right?

I'm not sure I fully understand this, but it does not seem quite right. One thing that might be confusing you is having your actions as "move to state". Whilst this is quite normal in many deterministic environments, especially board games, most RL formula and tutorials are written using separate state/action pairs, so whilst getting this straight in your mind, best to find another way to represent actions - e.g. move piece from X to Y, or place new piece at (I,J) . . . you can return to "change state from A1 to B2" as the representation later. This is actually more efficient and is called the after-state representation, but most of the literature will show state-action value functions.