In the book of Sutton and Barto (2018) Reinforcement Learning: An Introduction. The author defines the value function as.

$$v_{\pi}(\boldsymbol{s})=\mathbb{E}_{\boldsymbol{a}\,\sim\, \pi}\left[\sum_{k=0}^{\infty}\gamma^kR_{t+k+1}\,\bigg|\,\boldsymbol{s}_t=\boldsymbol{s} \right]$$

If $\boldsymbol{a}\in \mathcal{A}$ and $\boldsymbol{s}\in \mathcal{S}$ are continuous I would think by using Bellman's equation for the state-value function that this can be written as the integral

$$v_{\pi}(\boldsymbol{s})=\int_{\boldsymbol{a}\in\mathcal{A}}\pi\left(\boldsymbol{a}|\boldsymbol{s} \right)\int_{\boldsymbol{s}'\in \mathcal{S}}p(\boldsymbol{s}'|\boldsymbol{s},\boldsymbol{a})\left[R_{t+1}+\gamma v_{\pi}(\boldsymbol{s}')\right]d\boldsymbol{s'}d\boldsymbol{a}.$$

Is this correct?

Also without using Bellman's equation does the integral definition of the state-value function look like this?

$$v_{\pi}(\boldsymbol{s})=\int_{\boldsymbol{a}\in\mathcal{A}}\pi\left(\boldsymbol{a}|\boldsymbol{s} \right)\int_{\boldsymbol{s}'\in \mathcal{S}}p(\boldsymbol{s}'|\boldsymbol{s},\boldsymbol{a})\left[R_{t+1}+\gamma \left[\int_{\boldsymbol{a}'\in\mathcal{A}}\pi\left(\boldsymbol{a}'|\boldsymbol{s}' \right)\int_{\boldsymbol{s}''\in \mathcal{S}}p(\boldsymbol{s}''|\boldsymbol{s}',\boldsymbol{a}')\left[R_{t+2}+\gamma\left[\cdots\right] \right]d\boldsymbol{s''}d\boldsymbol{a}'\right] \right]d\boldsymbol{s'}d\boldsymbol{a}$$

Are my integral versions correct?


this can be written as the integral, is this correct?

Yes. Your derivations imply that we have assumed a deterministic reward given current state-action $(\boldsymbol{s},\boldsymbol{a})$. An stochastic reward model would be $p(\boldsymbol{s}', r|\boldsymbol{s},\boldsymbol{a})$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)

Are my integral versions correct?

Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial: $$f(n) = nf(n-1);f(0)=1$$ Which is unfolded as: $$f(n) = n [(n-1) [(n-2)[...]]]$$ However, the difference is that the index in Bellman equation is going forward since current value depends on future values not the previous ones.

| improve this answer | |
  • 1
    $\begingroup$ Thank you for the confirmation. I didn’t realize the part with the stochastic reward. $\endgroup$ – MachineLearner Mar 16 '19 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.