How to write out the definition of the value function for continous action and state space

Question

In the book of Sutton and Barto (2018) Reinforcement Learning: An Introduction. The author defines the value function as.

$$v_{\pi}(\boldsymbol{s})=\mathbb{E}_{\boldsymbol{a}\,\sim\, \pi}\left[\sum_{k=0}^{\infty}\gamma^kR_{t+k+1}\,\bigg|\,\boldsymbol{s}_t=\boldsymbol{s} \right]$$

If $\boldsymbol{a}\in \mathcal{A}$ and $\boldsymbol{s}\in \mathcal{S}$ are continuous I would think by using Bellman's equation for the state-value function that this can be written as the integral

$$v_{\pi}(\boldsymbol{s})=\int_{\boldsymbol{a}\in\mathcal{A}}\pi\left(\boldsymbol{a}|\boldsymbol{s} \right)\int_{\boldsymbol{s}'\in \mathcal{S}}p(\boldsymbol{s}'|\boldsymbol{s},\boldsymbol{a})\left[R_{t+1}+\gamma v_{\pi}(\boldsymbol{s}')\right]d\boldsymbol{s'}d\boldsymbol{a}.$$

Is this correct?

Also without using Bellman's equation does the integral definition of the state-value function look like this?

$$v_{\pi}(\boldsymbol{s})=\int_{\boldsymbol{a}\in\mathcal{A}}\pi\left(\boldsymbol{a}|\boldsymbol{s} \right)\int_{\boldsymbol{s}'\in \mathcal{S}}p(\boldsymbol{s}'|\boldsymbol{s},\boldsymbol{a})\left[R_{t+1}+\gamma \left[\int_{\boldsymbol{a}'\in\mathcal{A}}\pi\left(\boldsymbol{a}'|\boldsymbol{s}' \right)\int_{\boldsymbol{s}''\in \mathcal{S}}p(\boldsymbol{s}''|\boldsymbol{s}',\boldsymbol{a}')\left[R_{t+2}+\gamma\left[\cdots\right] \right]d\boldsymbol{s''}d\boldsymbol{a}'\right] \right]d\boldsymbol{s'}d\boldsymbol{a}$$

Are my integral versions correct?

Answer 1

this can be written as the integral, is this correct?

Yes. Your derivations imply that we have assumed a deterministic reward given current state-action $(\boldsymbol{s},\boldsymbol{a})$. An stochastic reward model would be $p(\boldsymbol{s}', r|\boldsymbol{s},\boldsymbol{a})$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)

Are my integral versions correct?

Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial: $$f(n) = nf(n-1);f(0)=1$$ Which is unfolded as: $$f(n) = n [(n-1) [(n-2)[...]]]$$ However, the difference is that the index in Bellman equation is going forward since current value depends on future values not the previous ones.