5
$\begingroup$

I cannot understand the way how algorithm Differential Semi-gradient Sarsa updates its estimated average reward $\bar{R}$. The algorithm I am looking at is from Sutton's text book Reinforcement Learning:An Introduction , section 10.3.

Why does not update $\bar{R}$ using the reward $R$ got at current step like $\bar{R} = (1-\beta)\bar{R}+\beta*R$ ? Since based on definition, $\bar{R}$ is the estimated average reward. I cannot understand the reason why the updating is like this: $\bar{R} = \bar{R}+\beta*\delta$, where $\delta$ is just the TD error. Why using TD error to update the average reward?

The following figure shows the algorithm. enter image description here

$\endgroup$
2
$\begingroup$

In a continuing task, TD error can increase w to infinity unless its expected value is zero. By subtracting the TD average estimation, the expected value of our update value is zero and w cannot go to infinity. TD error is a biased estimation of the average reward ( with a bias that goes to zero as the number of updates goes to infinity assuming that every state can be reached from every state since the average reward is independent of the state action combination with which we start).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.