It seems to me that the $V$ function can be easily expressed by the $Q$ function and thus the $V$ function seems to be superfluous to me. However, I'm new to reinforcement learning so I guess I got something wrong.

## Definitions

Q- and V-learning are in the context of [Markov Decision Processes](https://en.wikipedia.org/wiki/Markov_decision_process#Definition). A **MDP** is a 5-tuple $(S, A, P, R, \gamma)$ with

* $S$ is a set of states (typically finite)
* $A$ is a set of actions (typically finite)
* $P(s, s', a) = P(s_{t+1} = s' | s_t = s, a_t = a)$ is the probability to get from state $s$ to state $s'$ with action $a$.
* $R(s, s', a) \in \mathbb{R}$ is the immediate reward after going from state $s$ to state $s'$ with action $a$. (It seems to me that usually only $s'$ matters).
* $\gamma \in [0, 1]$ is called discount factor and determines if one focuses on immediate rewards ($\gamma = 0$), the total reward ($\gamma = 1$) or some trade-off.

A **policy $\pi$**, according to [Reinforcement Learning: An Introduction](https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node9.html) by Sutton and Barto is a function $\pi: S \rightarrow A$ (this could be probabilistic).

According to [Mario Martins slides](http://www.cs.upc.edu/~mmartin/Ag4-4x.pdf), the **$V$ function** is
$$V^\pi(s) = E_\pi \{R_t | s_t = s\} = E_\pi \{\sum_{k=0}^\infty \gamma^k r_{t+k+1} | s_t = s\}$$
and the **Q function** is
$$Q^\pi(s, a) = E_\pi \{R_t | s_t = s, a_t = a\} = E_\pi \{\sum_{k=0}^\infty \gamma^k r_{t+k+1} | s_t = s, a_t=a\}$$


## My thoughts

The $V$ function states what the expected overall value (not reward!) of a state $s$ under the policy $\pi$ is.

The $Q$ function states what the value of a state $s$ and an action $a$  under the policy $\pi$ is.

This means,
$$Q(s, \pi(s)) = V(s)$$

Right? So why do we have the value function at all? (I guess I mixed up something)