1
$\begingroup$

This question is from An introduction to RL Pages 48 and 49. This question may also be related to below question, although I am not sure: Cannot see what the "notation abuse" is, mentioned by author of book

On page 48, it is mentioned that p:S * R * S * A -> [0,1] is a deterministic function:

The dynamics function $p : \mathcal{S} \times \mathcal{R} \times \mathcal{S} \times \mathcal{A} \rightarrow [0, 1]$ is an ordinary deterministic function of four arguments.

However, on page 49, in equation 3.4, there is summation over r:

$$\sum_{s' \in \mathcal{S}}\sum_{r \in \mathcal{R}} p(s',r|s,a) = 1 ,\text{for all } s \in \mathcal{S}, a \in \mathcal{A}(s)$$

My question is, does this mean, it is possible that performing an action $a$ that takes us to state $s'$, could result in multiple rewards?

$\endgroup$
0
$\begingroup$

it is possible that performing an action $a$ that takes us to state $s′$, could result in multiple rewards?

Yes, that is true the general case that any $(s,a)$ pair can result in a range of results for both $s'$ and $r$. Also $s'$ and $r$ can vary independently, provided each ones distribution only depends on $(s,a)$. In practice $r$ often depends strongly on one or more of $s$, $a$ or $s'$ (if it depends on the latter, then it still depends in absolute terms on just $s$ and $a$ because $s'$ does - it is just that values of $s'$ and $r$ are allowed to correlate).

This does not affect the statement about $p(s',r|s,a)$ being deterministic. It is the probability of specific $s',r$ results occurring that should be deterministic and depend on $(s,a)$ in a Markov Decision Process.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.