# What are the advantages / disadvantages of off-policy RL vs on-policy RL?

There are various algorithms for reinforcment learning (RL). One way to group them is by "off-policy" and "on-policy". I've heard that SARSA is on-policy, while Q-Learning is off-policy.

I think they work as follows:

My questions are:

• How exactly is "on-policy RL" and "off-policy RL" defined?
• Please also let me know if there is an error in my pseudocode Jul 27 '16 at 14:35
• I think a good place to start to understand this would be this recent paper : jmlr.org/proceedings/papers/v32/silver14.pdf Aug 2 '16 at 6:47

This was answered in cross-validated and stackoverflow:

The reason that Q-learning is off-policy is that it updates its Q-values using the Q-value of the next state $$s′$$ and the greedy action $$a′$$. In other words, it estimates the return (total discounted future reward) for state-action pairs assuming a greedy policy were followed despite the fact that it's not following a greedy policy.

The reason that SARSA is on-policy is that it updates its Q-values using the Q-value of the next state $$s′$$ and the current policy's action $$a′′$$. It estimates the return for state-action pairs assuming the current policy continues to be followed.

These slides offer some insight on pros and cons of each one:

• On-policy methods:

• attempt to evaluate or improve the policy that is used to make decisions,
• often use soft action choice, i.e. $$\pi(s,a) >0, \forall a$$,
• commit to always exploring and try to find the best policy that still explores,
• may become trapped in local minima.
• Off-policy methods:

• evaluate one policy while following another, e.g. tries to evaluate the greedy policy while following a more exploratory scheme,
• the policy used for behaviour should be soft,
• policies may not be sufficiently similar,
• may be slower (only the part after the last exploration is reliable), but remains more flexible if alternative routes appear.

For reference, these are the formulations of Q-learning and SARSA from Sutton and Barto seminal book:

P.S.: I referenced and quoted the original answer from a different stackexchange site, as indicated in this meta question.