Skip to main content
Data Science

Return to Revisions

2 of 3
added 2 characters in body
user avatar
user avatar

How to implement a reward function which depends both on the next state and an action from current state?

There's the concept of "expected value of the next reward", often denoted as $\mathcal{R}^a_{ss'}$, and defined as

$$ \mathcal{R}^a_{ss'} = \mathbb{E}\left(r_{t+1} \mid s_t = s, a_t = a, s_{t+1} = s' \right) $$

which is the value we expect for the reward at the next time step, that is at time step $t+1$, given that action $a$ from state $s$ brings us to state $s'$.

But the italicized definition does not seem to be "consistent" with the Bellman equation for the value function, which is

\begin{align} V^\pi(s) &= \sum_{a \in \mathcal{A}(s)} \pi(s, a) \sum_{s' \in \mathcal{S^+}} \mathcal{P}^a_{ss'} \left( \mathcal{R}_{ss'}^a + \gamma V^\pi(s') \right) \end{align}

Why do I say this?

$\mathcal{R}_{ss'}^a$ depends on $s$, $a$ and $s'$, but in the Bellman equation above, specifically in the inner sum $\sum_{s' \in \mathcal{S^+}} \mathcal{P}^a_{ss'} \left( \mathcal{R}_{ss'}^a + \gamma V^\pi(s') \right)$, we are essentially iterating over all possible next states, $s'$, of $s$. But, if that's the case, then what's the purpose of making $\mathcal{R}_{ss'}^a$ depend on the action $a$? That is, if we already have the next state $s'$, then the action is useless, given that we are already taking into consideration the possible next states, indeed. In other words, $a$ and $s'$ may not be compatible.

For example, suppose we are in a certain state $s$, and we take action $a$ from there, and we end up in a state, say, $x$, which must be different to all next states $s'$ (in the inner summation), apart at most from one.

I hope you can see my point and my doubts, and I hope you can clarify them.

Furthermore, given this doubt, I am unable to implement a reward function. Also, I have seen an implementation of the reward function, that is

def reward(state, next_state, action):
    if GRID[state] != GOAL and GRID[next_state] == GOAL:
        return 10.0
    else:
        return 0

which actually does not use the action, but I don't understand why. Why should we use next_state and not the next state of state according to action (or vice-versa)? How could we use both next_state and action, if they may be "incompatible" in the sense I described above?

user10640