I'm working my way through the book Reinforcement Learning by Richar S. Sutton and Andrew G. Barto and I am stuck on the following question.

The value of a state depends on the the values of the actions possible in that state and on how likely each action is to be taken under the current policy. We can think of this in terms of a small backup diagram rooted at the state and considering each possible action:

enter image description here

Give the equation corresponding to this intuition and diagram for the value at the root node formula2 , in terms of the value at the expected leaf node, formula , given formula3 . This expectation depends on the policy, formula4 . Then give a second equation in which the expected value is written out explicitly in terms of formula5 such that no expected value notation appears in the equation.

I should mention that...

formula7 ...


formula6 = Probability of taking action a from state s

formula8 = Given any state s and a, the probability of each next state s'

formula9 = Expected reward given any state s, next state s, and action a

How can I re-evaluate this value function in the way that is asked?


1 Answer 1


In a discrete probability space, the expectation of a random variable (RV) is a sum over all possible values multiplied by their individual probabilities. Here, your RV is $q_\pi(s,a)$, with $s$ being fixed. Therefore you need to take the expecation with respect to the actions only:

$v_{\pi}(s) \stackrel{\text{1. task}}{=} \mathbb{E}_{a\sim \pi(s, a)}[q_{\pi}(s, a)] \stackrel{\text{2. task}}{=} \sum_\limits{a}\pi(s, a) q_{\pi}(s, a)$.

$\pi(s, a)$ represents the "individual probabilities".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.