In some resources, the belman equation is shown as below:

$v_{\pi}(s) = \sum\limits_{a}\pi(a|s)\sum\limits_{s',r}p(s',r|s,a)\big[r+\gamma v_{\pi}(s')\big] $

The thing that I confused is that, the $\pi$ and $p$ parts at the right hand side.

Since the probability part - $p(s',r|s,a)$- means that the probability of being at next state ($s'$), and since being at next state ($s'$) has to be done via following a specific action, the $p$ part also includes the probability of taking the specific actions inside it.

But then, why the $\pi(a|s)$ is written at the beginning of the equation? Why do we need it? Isn't the possibility of taking an action stated at the $p(s',r|s,a)$ part already?


1 Answer 1


$p(s', r | s, a)$ is the probability of arriving at state $s'$ and obtain reward $r$ given that the environment was in state $s$ and the agent took action $a$. Therefore, this probability is defined assuming action $a$ is taken. There is no probability of taking $a$ included there.

The probability of the agent taking an action is provided by the policy $\pi$, and that is why we need it in the equation.

You can think of the interaction of these two terms with the law of total probability: $p(A)=\sum _{n}p(A\mid B_{n})p(B_{n})$, where $p(B_{n})$ is analogous to $\pi(a|s)$ and $p(A\mid B_{n})$ is analogous to $p(s', r | s, a)$.

  • $\begingroup$ So, Are you suggesting that even if an action was selected under a probabilistic criteria which was represented by π, there is still a room for a different action? Why is that? Some sort of shock? $\endgroup$
    – datatech
    Feb 17, 2021 at 20:15
  • $\begingroup$ An action can be selected with a specific probability, defined by $\pi$. The $\sum_a$ adds over all possible actions. $\endgroup$
    – noe
    Feb 17, 2021 at 20:18
  • $\begingroup$ Well, sorry for making you bore. One more thing. I understand $\sum_a$ part. However, what I understand from your words is, you suggest that, even if an action is selected since it is more likely to be taken (say action $a_1$ which brings us to $s'$), it does not mean that we will reach the $s'$ for sure. There is still a probability to move some other state say $s''$ which has to be reached by taking another action say action $a_2$ which we haven't desired to select? $\endgroup$
    – datatech
    Feb 17, 2021 at 20:26
  • $\begingroup$ No. The value function (what you posted), estimates the value of some state $s$ by computing the rewards obtained reaching to all possible states $s'$ by all possible actions $a$. The value function does not deal with "selecting an action", just with the probability of doing so, the probability of reaching a state by taking an action, and the reward to be obtained doing so (and, recursively, the value of this new state). $\endgroup$
    – noe
    Feb 17, 2021 at 20:36
  • 1
    $\begingroup$ One thing that may not be clear to OP is that the sums are nested, which is how $a$ gets to be defined in the second sum. This is a standard notation for nested sums, but if the OP's background is more engineering than maths, it might be less familiar $\endgroup$ Feb 17, 2021 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.