I've seen in literature two different yet similar approaches when writing the value function in an MDP:

  1. $V_\pi(s)=\sum\limits_{a\in A}\pi(a|s)\sum\limits_{s'\in S}\sum\limits_{r\in R} Pr[s',r|s,a][r+\gamma V_\pi(s')]$
  2. $V_\pi(s)=\sum\limits_{a\in A}\pi(a|s)\sum\limits_{s'\in S} Pr[s'|s,a][r(s,a,s')+\gamma V_\pi(s')]$

Are those equivalent? what is the meaning of adding the sum over $r$?


1 Answer 1


It is the same one. I believe the Sutton's first version of the book had the first equation. The second version uses the second equation which carries the same meaning but simpler in writing. The reward will be a function of s, a and s'. For a fixed a, s there is exactly one reward corresponding to each s' possible. So the inner summation over r can be removed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.