3
$\begingroup$

In order to have an actor critic RL model there are two things to be satisfied .

  1. Value approximation function should converge to a local minimum

$$\sum_s d^{\pi}(s) \sum_a \pi(s,a)[Q^{\pi}(s,a) - f_w(s,a)]\frac{\partial f_w(s,a)}{\partial w} = 0$$

  1. The following condition should be satisfied with the parameterization

$$\frac{\partial f_w(s,a)}{\partial w} = \frac{\partial \pi(s,a)}{\partial \theta} \frac{1}{ \pi(s,a)} $$

So specifically how can we design a model to meet the second condition?

Update

here I want to highlight the value function approximation in actor-critic methods . We need to optimize the critic also as we did for the Q learning but following the on policy which is taking the TD error according to the actor. Here I will put the final equation of actor critic.

enter image description here

Here simply we can take the critic neural net's output as the state value function or the f above. So how to satisfy the condition 2 ?

Improve this question
$\endgroup$
1
  • $\begingroup$ May I ask why you are interested in on-policy critic form? Also how do you intend to train the critic's NN on-policy? $\endgroup$
    – Constantinos
    Commented Dec 9, 2017 at 20:40

1 Answer 1

Reset to default
3
$\begingroup$

There are many papers out there that deal with neural networks and RL. This blog will give a very good insight on a Policy Gradient Network: Deep RL with PG

Now for your question. You really need to be familiar on how we train a neural network. A simple one for classification. If you check the derivations and how the weights are getting updated things will be very clear to you on how you can implement the above.

I will describe it to you as simple as possible so you get the link. A Neural Network in a very broad sense consists of nested functions. The function that contains all the others is the one at your output layer. In case of Stochastic Policy Gradients this is your Boltzmann function. So your output layer will take all the previous layer's outputs and will pass them through the Boltzmann function. In the case of NN the parametrization comes from all previous layers.

The blog link I sent you describes a very nice and simple example of a vanilla Policy Gradient with NN (REINFORCE algorithm). By using the code (plus your understanding on feedforward networks) you will see that the gradients are multiplied by the reward. It is a very good exercise!

For Actor-Critic, you need in general a network performing PG (stochastic or deterministic) -- you Actor -- and a network that will give you the reward signal (like the simple case in the blog. However, for various reasons, instead of the actual reward we use another network that estimates the reward by performing Q-learning as in Deep-Q learning (minimizing square error between estimated reward and true reward).

Hope this helps!

Improve this answer
$\endgroup$
6
  • 3
    $\begingroup$ The relationship between this answer and equation 2 from the OP's question is a bit tenuous - basically you are saying the explanation is available from a paper linked from the blog/tutorial. I'm not sure I understand your third paragraph well enough to get whether it summarises things from the paper accurately. I am still stuck on questioning the assumption that $w$ and $\theta$ can even be guaranteed to have the same dimension since $f_w$ is always a scalar value, but $\pi$ can have any number of dimensions . . . I guess it would be nice to see some equation and terms in it explained. $\endgroup$
    – Neil Slater
    Commented Dec 8, 2017 at 18:25
  • $\begingroup$ well not really. It's not about the optimization of the neural net I asked. In actor critic methods we do maintain two neural networks . One is for the policy which is optimized with policy gradients and other one for the estimating the reward which is optimized with td error (value approximation can be simplified in to TD when we use a baseline reward function to reduce variance which is a value function ) , so in the second question there is a derivative need to be equivalent. How to fulfill that? $\endgroup$
    – Shamane Siriwardhana
    Commented Dec 9, 2017 at 17:32
  • 1
    $\begingroup$ The second condition is almost never true. There are approximations that can be made for certain classes of functions. Usually you end up having the $f_w=w^T \del{log\pi(\theta)}$. But with multiple nonlinear approximators such as NNs, theoretical performance guarantees are impossible. They used other methods such as target networks and slow updates to make the training stable. I am not sure Neil what do you mean about $\pi$ having infinite dimensions. The $f_w$ approximates the advantage function $A(s,a)$ and $\pi(s,a;\theta)$ is your policy. $\endgroup$
    – Constantinos
    Commented Dec 9, 2017 at 18:38
  • 1
    $\begingroup$ I mean that $\nabla_{w}f_w$ and $\nabla_{\theta} \text{log}(\pi)$ have trouble being equal unless they start with vector sizes match, which means both $w$ and $\theta$ must hold exactly same number of parameters. This is quite hard to arrange if $\pi$ has anything other than a single dimension, which it won't in many problems (only exception would be if there was a single real-valued action). BTW, the LaTex code you were looking for is \nabla $\endgroup$
    – Neil Slater
    Commented Dec 11, 2017 at 19:48
  • 1
    $\begingroup$ Yes, for the equality to hold we need equal size parameters. However as I stated above in the case of NNs these conditions do not hold. The OP makes the assumption that these conditions should hold for every parametrization which is not correct. And yes you do not have theoretical guarantees for AC with NNs -- at least to the extent of my knowledge. Thanks!I dont see how I can edit my comment... $\endgroup$
    – Constantinos
    Commented Dec 12, 2017 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.