Watching this video (11:30) that presents the simplest algorithm for reinforcement learning: Monte Carlo Policy Evaluation, which says in general:

The first time a sate is visited:

  1. increment N(s): N(s) = N(s) + 1
  2. increment total state's return function by current episode's return so far S(s) = S(s) + G_t

State's value is estimated by mean return over many episodes:

V(s) = S(s) / N(s)

by law of large numbers, V(s)-->V_true(S) as N(S)-->inf

My question is - should the environment always behave the same over different episodes, or can it change randomly, and we sill have the true value over a large enough number of episodes?


1 Answer 1


Yes, an assumption in the MDP formulation in RL is stationarity: the dynamics of the environment do not change over time (within or across episodes).

Note that this almost always false in the real world, but this assumption simplifies matters, so we have it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.