0
$\begingroup$

In DQN, we use;

$Target = r+\gamma v(s')$ equation to train (fit) our network. It is easy to understand since we use the $Target$ value as the dependent variable like we do in supervised learning. I.e. we can use codes in python to train the model like,

model.fit(state,target, verbose = 0)

where $r$ and $v(s')$ can be found by model prediction.

When it comes to A2C network, things becomes more complicated. Now we have got two networks. Actor and Ctitic. It is said, the Critic network is not different from the what it is done in DQN. Only difference is now we have got only one output neuron in the network. So, similarly, we calculate the $Target = r+\gamma v(s')$ after acting via sampled action from $\pi(a|s)$ distribution. And we train the model with model.fit(state,target, verbose = 0) in python as well.

However, in Actor case it is so confusing. Now, we have another neural network which takes states as input and gives probabilities as the output by using softmax activation function. It is cool. But the point I stuck is, the dependent variable to adjust the Actor network. So in python,

model2.fit(state,?,verbose=0)

What is the ? value to "supervise" the network to adjust the weights? and WHY?

In several resources I found the Advantage value which is nothing but the $Target - V(s)$. And also there is something called as the actor loss which is calculated by,

enter image description here

Why?

Thanks in advance!

$\endgroup$

1 Answer 1

2
$\begingroup$

The target will still be a form of a return estimation ($V(s_t)$, $Q(a_t,s_t)$, Advantage, n-step reward, etc). For example in your case the $Q_w$ that Critic estimated.

You will need to review a bit Policy Gradient methods in this order: PG Theorem, REINFORCE (Actor only method) then AC (Actor-Critic) and then A2C. I will give you a conceptual explanation, abstracted from the math behind. The general form of Policy Gradient is:

likelihood of action given state multiplied by a form of return: $log\pi_{\mathbf{\theta}}(a_t|s_t)\cdot R_t$

What this tells us is: maximize the likelihood of a specific action multiplied by the return. In classification we do know the correct class (action) but here this is not the case. The only learning signal comes from the reward. Therefore multiplying the likelihood of an action with a form of return for selecting that action, that will decrease/increase the probability of selecting that action again given at the current state. The PG theorem states that this is the direction to update the weights $\theta$ in order to maximize return.

In A2C, you can have various implementations: 2 separate networks or 1 network with 2 separate heads. Actor is responsible for learning the distribution of actions that maximize the return given state. Critic is responsible for estimating the return from current state (and action). Thus the loss function in A2C is usually the Policy Gradient loss plus the Mean Squared Error between expected return and observed return (plus entropy for exploration). Please refer to the original A2C paper for the equations. As you can see we have 2 main loss functions (one for each network (head) ).

The return, in any form, affects training in 2 ways:

  • Learning signal for the actor: Multiplies the likelihood of actions and by doing so increases/decreases the probability for selecting these actions. This will change the parameters of the policy in order for the policy to favor rewarding actions.
  • Learning signal for the critic: MSE target.

As you can see in a loose sense you are doing again supervised learning for both Actor and Critic but with classification loss for Actor and regression loss for Critic. The why this occurs comes from the Policy Gradient theorem that shows that in order to maximize the return following a parametrized policy we need to update the parameters towards the direction that maximize the PG loss function (for the Actor).

Please note that for various reasons using the return might not be that of a good idea. That's why you will see the PG loss in various forms ( e.g. policy likelihood multiplied by the advantage $A(t)$ or $V(s_t)-R_t$, $Q(a_t,s_t)$ etc)

$\endgroup$
8
  • $\begingroup$ Thank you for the informative answer. Here is the thing I don't understand in this equation : $log\pi_{\mathbf{\theta}}(a_t|s_t)\cdot R_t$. The $R_t$ part is coming from a specific next state which is reached by following a specific action. Since we take only one action in a step, we have only one reward. But we need to update the weights. For example say we have three action probs : 0.2 , 0.5, 0.3 . And we select 0.3 by sampling, and say we had an advantage -3. So you suggest that $-log(0.3) \cdot -3$ is the loss. Ok but what are the new three target probs then to train the network? $\endgroup$
    – datatech
    Apr 16, 2021 at 19:54
  • $\begingroup$ First of all $R_t$ is a return so it doesnt depend on action (or next state) only but future states as well. Why do you need 3 target probs to train the network? Could you tell me what you think you need in order to train the network (so i can understand your confusion)? $\endgroup$ Apr 17, 2021 at 6:02
  • $\begingroup$ $R_t$ is given by the environment after taking an action. So why is it independent from the action? The agent's action carries it to the next state which ends up with a reward. One more confusion :). Anyway, well, since ANN is a supervised learning algorithm, whenever I use it, inevitably I want to see Target variables. Since I did not see any in here, I got confused. Moreover, in this python code model2.fit(state,?,verbose=0) , python asks it for as well. It is more understandable the critic one or DQN. $\endgroup$
    – datatech
    Apr 17, 2021 at 6:32
  • $\begingroup$ From your question I sense that you are not familiar with basic RL and PG and you will get more and more confused especially because PG are quite more advanced in math. I will try to answer and hopefully help. $R_t$ is usually a discounted sum of immediate rewards $r(s,a)$ (or $r(s,a,s')$. I said that the return is not dependent ONLY on immediate action (and/or state). ANN is NOT a supervised learning algorithm. As I mentioned above the PG loss although it resembles a classification loss it is not as you do NOT know the true class, instead you get a reward. $\endgroup$ Apr 17, 2021 at 16:02
  • $\begingroup$ The reasons that PG loss ends up in this form are way different than the reasons that you end up minimizing cross entropy in classification problems. So basically you are searching for things that shouldn't exist :) $\endgroup$ Apr 17, 2021 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.