Skip to main content
added 326 characters in body
Source Link

Adding onto the other answers, note that sometimes it can be useful to express $V$ as an expectation of $Q$ over the distribution of actions according to the policy $\pi$:

$$ V^\pi(s) = \mathbb{E}_{a \sim \pi}[Q^\pi(s, a)] $$

Note that the subscript of the expectation operator there is using a shorthand notation. Sometimes you may see it written as $a \sim \pi(\, \cdot \,| s)$ to indicate that the expectation is across a random variable $a$ which is distributed according to the probability measure $\pi$ given the current state $s$.

This expectation expands out to the summation seen in other answers:

$$ V^\pi(s) = \sum_{a \in \mathcal{A}}\pi(a|s) Q^\pi(s, a) $$

Adding onto the other answers, note that sometimes it can be useful to express $V$ as an expectation of $Q$ over the distribution of actions according to the policy $\pi$:

$$ V^\pi(s) = \mathbb{E}_{a \sim \pi}[Q^\pi(s, a)] $$

which expands out to the summation seen in other answers:

$$ V^\pi(s) = \sum_{a \in \mathcal{A}}\pi(a|s) Q^\pi(s, a) $$

Adding onto the other answers, note that sometimes it can be useful to express $V$ as an expectation of $Q$ over the distribution of actions according to the policy $\pi$:

$$ V^\pi(s) = \mathbb{E}_{a \sim \pi}[Q^\pi(s, a)] $$

Note that the subscript of the expectation operator there is using a shorthand notation. Sometimes you may see it written as $a \sim \pi(\, \cdot \,| s)$ to indicate that the expectation is across a random variable $a$ which is distributed according to the probability measure $\pi$ given the current state $s$.

This expectation expands out to the summation seen in other answers:

$$ V^\pi(s) = \sum_{a \in \mathcal{A}}\pi(a|s) Q^\pi(s, a) $$

Source Link

Adding onto the other answers, note that sometimes it can be useful to express $V$ as an expectation of $Q$ over the distribution of actions according to the policy $\pi$:

$$ V^\pi(s) = \mathbb{E}_{a \sim \pi}[Q^\pi(s, a)] $$

which expands out to the summation seen in other answers:

$$ V^\pi(s) = \sum_{a \in \mathcal{A}}\pi(a|s) Q^\pi(s, a) $$