this can be written as the integral, is this correct?

Yes.

Are my integral versions correct?

Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial: $$f(n) = nf(n-1);f(0)=1$$ Which is unfolded as: $$f(n) = n [(n-1) [(n-2)[...]]]$$ However, the difference is that the index in Bellman equation is going forward since current value depends on future values not the previous ones.

Side note

Your derivations imply that we have assumed a deterministic reward given current state-action $(\boldsymbol{s},\boldsymbol{a})$. An stochastic reward model would be $p(\boldsymbol{s}', r|\boldsymbol{s},\boldsymbol{a})$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)

this can be written as the integral, is this correct?

Yes. Your derivations imply that we have assumed a deterministic reward given current state-action $(\boldsymbol{s},\boldsymbol{a})$. An stochastic reward model would be $p(\boldsymbol{s}', r|\boldsymbol{s},\boldsymbol{a})$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)

Are my integral versions correct?

Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial: $$f(n) = nf(n-1);f(0)=1$$ Which is unfolded as: $$f(n) = n [(n-1) [(n-2)[...]]]$$ However, the difference is that the index in Bellman equation is going forward since current value depends on future values not the previous ones.

this can be written as the integral, is this correct?

Yes.

Are my integral versions correct?

Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial: $$f(n) = nf(n-1);f(0)=1$$ Which is unfolded as: $$f(n) = n [(n-1) [(n-2)[...]]]$$ However, the difference is that the index in Bellman equation is going forward since current value depends on future values not the previous ones.

this can be written as the integral, is this correct?

Yes.

Are my integral versions correct?

Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial: $$f(n) = nf(n-1);f(0)=1$$ Which is unfolded as: $$f(n) = n [(n-1) [(n-2)[...]]]$$ However, the difference is that the index in Bellman equation is going forward since current value depends on future values not the previous ones.

Side note

Your derivations imply that we have assumed a deterministic reward given current state-action $(\boldsymbol{s},\boldsymbol{a})$. An stochastic reward model would be $p(\boldsymbol{s}', r|\boldsymbol{s},\boldsymbol{a})$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)