3
$\begingroup$

In Chapter 7.3 of Reinforcement Learning: An Introduction by Sutton and Barto, the off-policy pseudocode has the following update equation for $Q$:

  1. Compute importance sampling ratio: $$ \rho \leftarrow \prod^{\min(\tau+n-1, T-1)}_{i=\tau+1} \frac{\pi(A_i \mid S_i)}{b(A_i \mid S_i)} $$

  2. Compute truncated return $$ G \leftarrow \sum^{\min(\tau+n, T)}_{i = \tau+1} \gamma^{i-\tau-1} R_i $$

  3. Compute discounted estimate:

$$ \text{If } \tau+n < T, G \leftarrow G + \gamma^n Q(S_{\tau+n}, A_{\tau+n}) $$

  1. Update $Q$

$$ Q(S_\tau, A_\tau) \leftarrow Q(S_\tau, A_\tau) + \alpha \rho [G - Q(S_\tau, A_\tau)] $$

($\tau$ is the time whose estimate is being updated, $t$ is current timestep, $n$ is $n$-step return, and $T$ is termination timestep)

I believe that the importance sampling ratio $\rho$ should only be multiplied to $G$ and not to $Q$, since $Q$ is for target policy $\pi$ and $G$ is from behavior policy $b$. In other words, this is how I see this equation:

$$ Q_\pi \leftarrow Q_\pi + \alpha \rho_{b \to \pi} [G_b - Q_\pi]$$

Am I correct in thinking that this is a typo, or did I miss something?

Thank you for your help!

$\endgroup$
3
$\begingroup$

It seems like someone else asked themselves the same question on Cross Validated. It makes sense for the ratio to be multiplied to G only, since it's only G that uses the behavior policy to sample the next n rewards.

$\endgroup$
1
  • $\begingroup$ Didn't see that! Thank you for the reference. $\endgroup$ Sep 3 '18 at 1:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.