1
$\begingroup$

Slide 30 of these lecture notes for reinforcement learning contains this "Bellman equation":

$$ Q^*(s, a) = \mathbb{E}_{s'\sim\varepsilon} \left[ r + \gamma \max_{a'} Q^*(s', a') \mid s, a \right]$$

Everything else makes sense to me, but what does the "$\sim\varepsilon$" part mean?

Improve this question
$\endgroup$
3
  • $\begingroup$ See stats.stackexchange.com/q/41306/232706. But I don't find any definition for $\varepsilon$ in the earlier slides, and I'm not familiar enough with the equation to say from personal knowledge. It should mean something about "next state" and transition probabilities?? $\endgroup$
    – Ben Reiniger ♦
    Commented Jul 7, 2022 at 14:01
  • 2
    $\begingroup$ It means that $s'$ is distributed as $\epsilon$, ie: the values $s'$ are taken from $\epsilon$. However, there's no definition of $\epsilon$ in the slides you provided, so it's difficult to know what it means. Maybe it's a standard notation in RL, can you provide more context? $\endgroup$
    – alexmolas
    Commented Jul 7, 2022 at 14:06
  • $\begingroup$ I assume the definition of epsilon is kind of implicit. That helped me thanks. $\endgroup$
    – samuelnihoul
    Commented Jul 7, 2022 at 14:35

2 Answers 2

Reset to default
2
$\begingroup$

It is not an $\epsilon$, but an $\mathcal{E}$, which probably stands for environment. The more common notation in RL is

$$Q^*(s,a)=\mathbb{E}_{s'\sim p(\cdot\mid s,a)}[r(s,a)+\gamma \max_{a'}Q^*(s',a')\mid s,a]. $$

What it means is that we take action $a$ in state $s$. Then, the environment returns a state $s'$. So, it is an expectation over all possible new states, when taking action $a$ in state $s$.

Improve this answer
$\endgroup$
-1
$\begingroup$

Symbolizes the expected error.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Some kind of justification would make this kind of short answer better. But in this case, I also think this is wrong: see my comment on the Question. $\endgroup$
    – Ben Reiniger ♦
    Commented Jul 7, 2022 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.