According to my understanding, in Deep Q-learning, in order to train the NN, the agent stores experiences in a buffer (Replayed Memory), and each experience contains:

e = <s,a,r,s'>

When we train the model, we input the s in the target NN, and choose the Q value of action a. Then we input the s' in the prediction network and take the max Q value (then we calculate the loss using the reward r...).

My problem is that I don't understand why do we store the action a and take its Q value in the target network, if we always take the action with the max Q value.

Doesn't the data of action a redundant?


1 Answer 1


In a very general form, temporal difference (TD) learning is based on the idea that a value (typically a state value or action value) at time $t$ is related to the value at time $t+n$, and this can be used to improve estimates of the value at time $t$.

In single-step TD learning using action values, the values that get related, and are used to drive the update are $Q(s_t, a_t)$ and $Q(s_{t+1}, a_{t+1})$. This applies to Q learning and SARSA - in both of these, in order to process an update, you need to select which parameters the estimate is being updated for i.e. $s_t, a_t$ (for both SARSA and Q-learning) and which parameters are used to calculate the TD target. It is these second set of parameters where selecting $a_{t+1}$ differs between SARSA and Q-learning. SARSA should use the action that was actually taken, and Q-learning can use the current greedy action.

The update is always for a specific pair $s_t, a_t$ - you are estimating the value of an observed state/action pair, based on the immediate reward and state transition seen after it. You can alter that estimate based on a different target policy, but you cannot alter what it is an estimate for - swapping $a_t$ for some other action means you don't know what the immediate reward and next state should be.

So, the action $a_t$ in the experience replay table selects which estimate you are updating. In Q learning you don't need to store the next action $a_{t+1}$ because you will use the greedy action instead to calculate the update.

As an aside, in SARSA you cannot normally use it with an experience replay table because the old experiences are all off-policy. However, you could store a bunch of experiences without updating, then use them in a single batch update. In that case then you would store both $a_t$ and $a_{t+1}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.