# Confusion about the Bellman Equation

In some resources, the belman equation is shown as below:

$$v_{\pi}(s) = \sum\limits_{a}\pi(a|s)\sum\limits_{s',r}p(s',r|s,a)\big[r+\gamma v_{\pi}(s')\big]$$

The thing that I confused is that, the $$\pi$$ and $$p$$ parts at the right hand side.

Since the probability part - $$p(s',r|s,a)$$- means that the probability of being at next state ($$s'$$), and since being at next state ($$s'$$) has to be done via following a specific action, the $$p$$ part also includes the probability of taking the specific actions inside it.

But then, why the $$\pi(a|s)$$ is written at the beginning of the equation? Why do we need it? Isn't the possibility of taking an action stated at the $$p(s',r|s,a)$$ part already?

$$p(s', r | s, a)$$ is the probability of arriving at state $$s'$$ and obtain reward $$r$$ given that the environment was in state $$s$$ and the agent took action $$a$$. Therefore, this probability is defined assuming action $$a$$ is taken. There is no probability of taking $$a$$ included there.
The probability of the agent taking an action is provided by the policy $$\pi$$, and that is why we need it in the equation.
You can think of the interaction of these two terms with the law of total probability: $$p(A)=\sum _{n}p(A\mid B_{n})p(B_{n})$$, where $$p(B_{n})$$ is analogous to $$\pi(a|s)$$ and $$p(A\mid B_{n})$$ is analogous to $$p(s', r | s, a)$$.
• An action can be selected with a specific probability, defined by $\pi$. The $\sum_a$ adds over all possible actions. – noe Feb 17 at 20:18
• Well, sorry for making you bore. One more thing. I understand $\sum_a$ part. However, what I understand from your words is, you suggest that, even if an action is selected since it is more likely to be taken (say action $a_1$ which brings us to $s'$), it does not mean that we will reach the $s'$ for sure. There is still a probability to move some other state say $s''$ which has to be reached by taking another action say action $a_2$ which we haven't desired to select? – datatech Feb 17 at 20:26
• No. The value function (what you posted), estimates the value of some state $s$ by computing the rewards obtained reaching to all possible states $s'$ by all possible actions $a$. The value function does not deal with "selecting an action", just with the probability of doing so, the probability of reaching a state by taking an action, and the reward to be obtained doing so (and, recursively, the value of this new state). – noe Feb 17 at 20:36
• One thing that may not be clear to OP is that the sums are nested, which is how $a$ gets to be defined in the second sum. This is a standard notation for nested sums, but if the OP's background is more engineering than maths, it might be less familiar – Neil Slater Feb 17 at 20:44