What is Compatible Function Approximation theorem in reinforcement learning?

Question

I am following David Silver's RL course. In the policy gradient section, I found this slide that I would like have an explanation of.

What are these two conditions?

What is the logic behind the first derivative equality? Is it just that we assume these two derivatives should be equal since there should be some kind of connection with direction of value function approximation gradient and our policy like-hood?

Then what is the epsilon value? What is this mean square value? Q value with w parameters means the Q values function new approximation. And the other Q value is the value got by following our policy right? So is this OFF policy value approximation or ON policy? I think this is on policy since normally in the Q value update we take the max.

Answer 1

There are lots of questions but I will try to answer in a way that might clear things up for you and also give you some guidance. Please note that the proofs for your questions involve lots of math operations so instead I will provide you with references.

Your main reference is the paper from Sutton PG Methods with Function Approximation. I highly recommend you to read the paper a couple of times (or even more!) and do some search in the relevant literature when you will be familiar with the main objectives, notation and math around the general approach of the methods. PG Methods are not easy to get a grasp of them mainly because of their sampling nature, notation and discrete/continuous math involved.

PG Methods satisfy (or at least should) the PG theorem (eq. 2 from the paper). An interesting approach would be to substitute the true $Q^\pi (s,a)$ by some approximate function ($f_w$ in the paper, $Q_w$ in your question). Now, we are wondering what conditions should be satisfied by that proposed approximation in order to satisfy the PG Theorem.

The first thing you notice is that a natural choice for updating the parameters $w$ is to update them towards the direction that minimizes the mean squared error of the exact $Q^\pi (s,a)$ with the function approximation. In your question this is the $\epsilon$. In such a scenario the exact $Q^\pi (s,a)$ is estimated using unbiased samples such as $r_t$. This is explained in detail in Part 2 of the paper.

For the PG theorem to hold (proof consists of the 3 lines before Part 3) the grad of your approximate function should satisfy the compatibility condition. To sum up we started from PG theorem and we found a suitable family of function approximators for our action-value function that the PG theorem holds. In Part 3 you can see an example of a compatible function. From this of course you can use even non-linear approximators such as NNs.

A clarification on on/off-policy: The David Silver's slide that you posted here has to do with theoretical guarantees and has nothing to do with an actual RL algorithm. By the way Q-learning algorithm in which you use the $max_{a'}{Q(s',a')}$ is OFF-policy as you you don't actually use for updates the ongoing policy.

Hope this helps!