# Confusion about the Bellman Equation

In some resources, the belman equation is shown as below:

$$v_{\pi}(s) = \sum\limits_{a}\pi(a|s)\sum\limits_{s',r}p(s',r|s,a)\big[r+\gamma v_{\pi}(s')\big]$$

The thing that I confused is that, the $$\pi$$ and $$p$$ parts at the right hand side.

Since the probability part - $$p(s',r|s,a)$$- means that the probability of being at next state ($$s'$$), and since being at next state ($$s'$$) has to be done via following a specific action, the $$p$$ part also includes the probability of taking the specific actions inside it.

But then, why the $$\pi(a|s)$$ is written at the beginning of the equation? Why do we need it? Isn't the possibility of taking an action stated at the $$p(s',r|s,a)$$ part already?

$$p(s', r | s, a)$$ is the probability of arriving at state $$s'$$ and obtain reward $$r$$ given that the environment was in state $$s$$ and the agent took action $$a$$. Therefore, this probability is defined assuming action $$a$$ is taken. There is no probability of taking $$a$$ included there.

The probability of the agent taking an action is provided by the policy $$\pi$$, and that is why we need it in the equation.

You can think of the interaction of these two terms with the law of total probability: $$p(A)=\sum _{n}p(A\mid B_{n})p(B_{n})$$, where $$p(B_{n})$$ is analogous to $$\pi(a|s)$$ and $$p(A\mid B_{n})$$ is analogous to $$p(s', r | s, a)$$.

• So, Are you suggesting that even if an action was selected under a probabilistic criteria which was represented by π, there is still a room for a different action? Why is that? Some sort of shock? Feb 17, 2021 at 20:15
• An action can be selected with a specific probability, defined by $\pi$. The $\sum_a$ adds over all possible actions.
– noe
Feb 17, 2021 at 20:18
• Well, sorry for making you bore. One more thing. I understand $\sum_a$ part. However, what I understand from your words is, you suggest that, even if an action is selected since it is more likely to be taken (say action $a_1$ which brings us to $s'$), it does not mean that we will reach the $s'$ for sure. There is still a probability to move some other state say $s''$ which has to be reached by taking another action say action $a_2$ which we haven't desired to select? Feb 17, 2021 at 20:26
• No. The value function (what you posted), estimates the value of some state $s$ by computing the rewards obtained reaching to all possible states $s'$ by all possible actions $a$. The value function does not deal with "selecting an action", just with the probability of doing so, the probability of reaching a state by taking an action, and the reward to be obtained doing so (and, recursively, the value of this new state).
– noe
Feb 17, 2021 at 20:36
• One thing that may not be clear to OP is that the sums are nested, which is how $a$ gets to be defined in the second sum. This is a standard notation for nested sums, but if the OP's background is more engineering than maths, it might be less familiar Feb 17, 2021 at 20:44