Deep Q-Learning: How are network parameters updated, and why consider episodes in the first place?

Question

I'm trying to wrap my head around the implementation of deep $Q$-learning, and why we even consider episodes in the first place. The usual set-up is that we initialize some starting state $s_0$, then let the agent run through the environment (following for example an epsilon-greedy policy with respect to the Q-values generate by a policy/online network) to create some set of episodes $e_t = (s_t, a_t, R_{t+1}, s_{t+1})$, where $s_t$ is the state at time $t$, $a_t$ is the action chosen at $s_t, R_{t+1}$ is the reward for taking action $a_t$ at state $s_t$, and $s_{t+1}$ is the state obtained from taking action $a_t$ at $s_t$.

In what follows, I denote the feedforward of state $s$ through the online net by $ff_o(s)$, and through the target net by $ff_t(s)$. Then, by construction, $$ff_o(s) = \begin{bmatrix} Q(s, a^1) \\ \vdots \\ Q(s, a^n)\end{bmatrix}$$ where $a^1, \dots, a^n$ denote the possible actions that the agent can take (ordered in some pre-determined way, and for some integer $n$). Suppose that $s'$ is the state obtained by taking action $a$ from state $s$. Then, we define $Q_{\max}(s, a) = \max ff_t(s')$. Our aim is to update the network parameters so that $Q(s, a)$ moves closer to $R(s, a) + \gamma Q_{\max}(s, a)$ for every state-action pair in our episodes, where $\gamma$ is the discount factor.

We define the loss function for our network by $$l(s, y) = \|ff_o(s) - y\|^2$$ where $y$ is the target output for input $s$, and $\|\cdot\|$ denotes the standard vector norm in $\mathbb{R}^n$. The training data takes the form $(s_t, y_t)$, where $s_t$ is the initial state in the episode $e_t$ and $y_t$ is the target output of the online network for input $s_t$. How should we choose the target outputs $y_t$?

I see two potential approaches.

1. Suppose that $a_t = a^k$ in our ordering of the actions. Let $x[i]$ denote the $i^{\text{th}}$ component of the vector $x \in \mathbb{R}^n$. Then, we choose $$y_t[i] = \begin{cases} R(s_t,a_t) + \gamma Q_{\max}(s_t, a_t), &\text{if } i = k, \\ ff_o(s_t)[i], &\text{if } i \ne k \end{cases}.$$ Effectively, this says that we choose the component of $y_t$ that corresponds to the action $a_t$ of the episode $e_t$ as the right-hand side of the Bellman equation. Otherwise, we choose the $Q$-values as they currently are. In that sense, our loss function reduces to $(Q(s_t, a_t) - R_{t+1} - \gamma Q_{\max}(s_t, a_t))^2$. Is this the typical strategy?
2. In the case that the reward function $R(s, a)$ is given, we choose $y[i] = R(s, a^i) + \gamma Q_{\max}(s, a^i)$ for all $i$. That is, we calculate the RHS of the Bellman equation over all possible actions that we can take from the state $s$. In this sense, the episodes only enter as some way to sample states $s_t$ from which the corresponding target outputs $y_t$ can be generated. Is there a problem with this approach? What stops us from just randomly sampling states in our environment and generating training data this way, instead of letting the agent explore in episodes?

Answer 1

The neural network's weights are updated to reduce the difference between the predicted Q-value $Q(s,a;θ)$ and the target Q-value $y$. Since we're only interested in the action that was actually taken, the gradient for actions not taken is zero, and they do not affect the update. This selective updating means that the neural network learns the value of the action that was chosen, without interfering with the values of other actions.

Your loss is dependent only on actions that the agent selected. Since the loss only depends on the Q-value for the action that was actually taken, the partial derivatives (gradients) of the loss with respect to the Q-values of the other actions are zero. This is why the parameters of the network are only updated in relation to the action that was actually taken and the Q-values for the other actions remain unchanged for this particular training instance.