Evaluating value functions in RL

Question

I'm working my way through the book Reinforcement Learning by Richar S. Sutton and Andrew G. Barto and I am stuck on the following question.

The value of a state depends on the the values of the actions possible in that state and on how likely each action is to be taken under the current policy. We can think of this in terms of a small backup diagram rooted at the state and considering each possible action:

Give the equation corresponding to this intuition and diagram for the value at the root node , in terms of the value at the expected leaf node, , given . This expectation depends on the policy, . Then give a second equation in which the expected value is written out explicitly in terms of such that no expected value notation appears in the equation.

I should mention that...

...

Where...

= Probability of taking action a from state s

= Given any state s and a, the probability of each next state s'

= Expected reward given any state s, next state s, and action a

How can I re-evaluate this value function in the way that is asked?

Answer 1

In a discrete probability space, the expectation of a random variable (RV) is a sum over all possible values multiplied by their individual probabilities. Here, your RV is $q_\pi(s,a)$, with $s$ being fixed. Therefore you need to take the expecation with respect to the actions only:

$v_{\pi}(s) \stackrel{\text{1. task}}{=} \mathbb{E}_{a\sim \pi(s, a)}[q_{\pi}(s, a)] \stackrel{\text{2. task}}{=} \sum_\limits{a}\pi(s, a) q_{\pi}(s, a)$.

$\pi(s, a)$ represents the "individual probabilities".