Is the neural network in DQN used to learn like a supervised model?

Question

Is the neural network in DQN used to learn like a supervised model?

Answer 1

Is the neural network in DQN used to learn like a supervised model?

Yes.

In DQN, the neural network is used as a function approximator to learn the action value function $Q(s,a)$ - the neural network approximation to it is sometimes noted $\hat{q}(s,a,\theta)$ to show that this is an ongoing, approximate estimate, and that the values that it outputs depend on the NN's learnable parameters (weights and biases).

Why is this "supervised learning"? That is because in order to improve an estimate for Q, the network is presented with labelled data. In DQN this is specifically a mini-batch drawn from experience replay memory, but in all cases in RL, the format of each row of data is the same: A representation of the current state (or state+action sometimes), and a "ground truth" target value to learn to associate with that input. This learning by correct example is exactly supervised learning.

Now, there are some complications:

• The "ground truth" is built by the RL algorithm on-demand, and equals the underlying real value $q_{\pi}(s,a)$ only in expectation over time. However, that is fine for any statistics-based learning algorithm that will learn a mean value.

• The expected "ground truth" changes over time as RL improves the policy, so higher Q values will be expected, and the old lower ones need to be forgotten. This is why RL algorithms build the "ground truth" on demand - for single-step Q-learning, as used in basic DQN that is the $r + \gamma \text{max}_{a'}\hat{q}(s', a')$ formula that you may see in explanations, and is called the TD Target. That means that RL algorithms prefer to work with online algorithms that are biased towards most-recent data and forget older data. Neural networks and linear regression trained by gradient descent both fit this requirement.

An aside: In RL, to avoid using the term model twice, the term "function approximator" is used, where supervised learning might call the same thing a "model". That is because RL already uses the term model to refer to models of the environment (e.g. the rules of a game, or all the known transition probabilities and rewards).