3
$\begingroup$

I'm currently reading Hands on Machine Learning with Scikit-Learn & Tensorflow, and I'm wondering why does Q-learning require an actor model and a critic model to learn?

On page 465, it states:

As we will see, the training algorithm we will use requires two DQNs with the same archicture (but different parameters): one will be used to drive Ms. Pac-Man during training (the actor), and the other will watch the actor and learn from its trials and errors (the critic).

Is this a typical Q-learning implementation? If not, what is?

$\endgroup$
  • 1
    $\begingroup$ It doesn't. Actor-critic is a separate learning algorithm to Q-learning, and Q-learning should have an "action-value function" which is similar conceptually to the critic in Actor-critic, but not the same. Could you provide a link or quote the text? It might be confusing text on the tutorial, or a misunderstanding. $\endgroup$ – Neil Slater May 10 '18 at 13:03
  • $\begingroup$ Yeah I was afraid that I may have confused one way of using Q-learning as the common definition of Q-learning. I will add a quote when it first mentions actor and critic models. $\endgroup$ – AshleyS May 12 '18 at 4:52
0
$\begingroup$

The book you are reading is being somewhat lax with terms. It uses the terms "actor" and "critic", but there is another algorithm called actor-critic which is very popular recently and is quite different from Q learning. Actor-critic does have two function estimators with the roles suggested in the quote. Q-learning has one such estimator*.

I have looked at the chapter in more detail, and where it says:

one will be used to drive Ms. Pac-Man during training (the actor), and the other will watch the actor and learn from its trials and errors (the critic).

I would substitute:

one will be used learn from current actions, and the other will remember results from some time steps ago in order to estimate the values for next actions.

This is not something that's inherently part of Q-learning, but it is part of DQN's adjustments when combining Q-learning with neural networks. Both experience replay and having two copies of the learning network (one a temporarily "frozen" version of the other) are important for stabilising the learning algorithm. Without them it can become numerically unstable.

Is this a typical Q-learning implementation?

It's a typical implementation of basic DQN, which is how many people nowadays would implement Q-learning with neural networks.

You can ignore the references to "actor" and "critic". Instead it is easier to consider that there is just one "action value" network, and you keep an old copy of it around to help with stability.


* Generally in RL, the term "model" is reserved for a model of the environment - which neither Q-learning nor actor-critic provide. So you will also read that Q-learning is a "model free" algorithm. For the rest of the book, you will have seen "model" to refer to any statistical learning algorithm (or the architecture and learned parameters) . . . what you will see in RL texts is the careful use of "function estimator" or other terms for networks which learn something else other than how the environment behaves.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.