The value of a state $s$ under a certain policy $\pi$, $V^\pi(s)$, is defined as the "expected return" starting from state $s$. More precisely, it is defined as
$$ V^\pi(s) = \mathbb{E}\left(R_t \mid s_t = s \right) $$
where $R_t$ can be defined as
$$ \sum_{k=0}^\infty \gamma^k r_{t+k+1} $$
which is a sum of "discounted" rewards after time $t$, i.e. starting from time $t+1$.
$V^\pi(s)$ can also be interpreted, even more precisely, as the expected cumulative future discounted reward. This denotation contains all the words which refer to specific parts of the formula above, where
- "Expected" refers to the "expected value"
- "Cumulative" refers to the summation
- "Future" refers to the fact that it's an expected value of a future quantity with respect to the present quantity, i.e. $s_t = s$.
- "Discounted" refers to the "gamma" factor, which is a way to adjust the importance of how much we value rewards at future time steps, i.e. starting from $t + 1$.
- "Reward" refers to the main quantity of interested, i.e. the reward received from the environment.
Meanwhile, I've heard the term "expected reward", but I am not sure if it refers to the same concept or not, that is if "expected reward" or "expected return" are the same thing or not.
I know there's also the concept of "expected value of the next reward", often denoted as $\mathcal{R}^a_{ss'}$, and defined as
$$ \mathcal{R}^a_{ss'} = \mathbb{E}\left(r_{t+1} \mid s_t = s, a_t = a, s_{t+1} = s' \right) $$
which, again, is the value we expect for the reward at the next time step, that is at time step $t+1$, given that action $a$ from state $s$ brings us to state $s'$.
Is the "expected reward" actually $\mathcal{R}^a_{ss'}$ instead of $V^\pi(s)$?