Skip to main content
1 of 2

Adding onto the other answers, note that sometimes it can be useful to express $V$ as an expectation of $Q$ over the distribution of actions according to the policy $\pi$:

$$ V^\pi(s) = \mathbb{E}_{a \sim \pi}[Q^\pi(s, a)] $$

which expands out to the summation seen in other answers:

$$ V^\pi(s) = \sum_{a \in \mathcal{A}}\pi(a|s) Q^\pi(s, a) $$