I am trying to solve some questions about a MRP (i.e. a Markov Decision process with only one possible action at each state). The setup is as follows:
There are two states ($a$ and $b$) stepping to $a$ is terminal.
All rewards are zero, discount for stepping from $b \to b$ is $1$, and discount for all other steps is zero
The possible scenarios are: $a\to b$ (probability $1$), $b\to a$ (probability $p$) and $b\to b$ (probability $1-p$).
The first question I have is whether or not its true that the optimal values for each state here are zero? If not how did you derive this?
Second question I have is if we want a parameter $\lambda$ and have a feature $\phi$ such that
$\lambda \times \phi(a)$ and $\lambda \times \phi(b)$
approximate the optimal values at the states a and b and we attempt to approximate such a $\lambda$ by means of TD(0) starting with $\lambda_0 = 1$ how can I find the expected value of $\lambda$ after one episode of updating in terms of $p$? ($E[\lambda_T]$ where $T$ is a random number representing the duration of the episode)
By an episode I mean if we go back from state b to a the episode is over.