4
$\begingroup$

I am trying to understand the implementation of the actor critic reinforcement learning algorithm. According to this, there should be just one neural network with two heads for the action probabilities and the state values. It is also true in their tensorflow implementation here.

However, if I refer to the tensorflow implementations such as this and this, they use two nn and update them separately.

So, my question is what is the correct way to go about the neural network for the actor critic algorithm?

$\endgroup$
2
$\begingroup$

Both architectures work. It is probably more common to use two separate networks for simpler problems, and a combined network for a more complex problem, such as one involving machine vision.

In general, you need the two functions - policy and value - to be separate. There is no reason to expect them to have too much in common, in terms of the overall mapping from state to their output. However, if the state requires a lot of non-linear interpretation to get meaningful features, such as image, audio or video input, then it could be an overall benefit if the two functions share the lower-level feature representations. Not only will this encourage better generic learning of the low-level features (because they are effectively being trained with twice the data per time step compared to if they were separate), but the calculations should be faster too.

For a similar reason, if the agent works from natural images, you could use either pre-processed features from upper layers of some model trained on ImageNet, or you could start the network initialised with the first layers of such a model. This might apply for other RL agents, such as DQN too - if this works, then clearly sharing that part of the network when you have more than one function to work on could work also.

If your state data is simpler, e.g. a few positions and velocities, or one/both policy and value have a simple relationship to state, then a joint network may be less useful.

When the policy and value functions are in a shared network, they may still both have more than one dedicated layer, as it is expected that there is not any simple linear relationship between the two functions (a NN could learn this by itself, but the assumption is good enough that it may as well be hard-coded by splitting the network into two branches with maybe a couple of hidden layers each before output).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.