1
$\begingroup$

I'm trying to understand the difference between target-values and action-values in Deep Q Networks.

From what I understand, action-value tries to approximate the reward of a given action (at some state). The target-value is also an approximate of the reward. How are they different?

Reference
- https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf

$\endgroup$
2
$\begingroup$

Action value and target value in DQN refer to the same thing in terms of what they measure, but are obtained and used in different ways.

An action value does not approximate the reward of a given action, but represents the related concept of expected return - the expected discounted sum of future rewards when taking action $a$ in state $s$. You see it very often associated with the action value function $q(s,a)$.

Action values are not inherently approximated, they are more conceptual than that. However, in practice in Reinforcement Learning (RL), the action value function must be learned, and some form of approximation used based on observed data. When an action value function it is approximated by a neural network during learning you may see it written $\hat{q}(s,a,\theta)$ which makes explicit that the NN is approximating some "true" function that you don't know, and that it is parametrised by $\theta$, the neural network weights and biases.

During learning, the agent takes actions and observes resulting states and rewards. It must use these observations to improve its estimates of values. To do so in DQN, the agent constructs a temporal difference (TD) target - for single-step Q-learning this is $G_{t:t+1} = r_{t+1} + \gamma\text{max}_{a'}\hat{q}(s_{t+1},a',\theta)$. This is the "target value" and there are multiple ways to construct it, giving rise to variants of RL algorithms.

The value $G_{t:t+1}$ is also an estimate of expected return. Technically it is a sampled estimate, which may vary and may be biased (due to starting conditions of the neural network). However, it includes some real experience, so may be used to improve the neural network through training. You can use it like ground truth associated with the state and action, to train the neural network.

The TD target or "target value" gets its name because by updating a Q table or training a NN with it as a ground truth, the estimator will output values in future closer to the supplied value. The estimator "gets closer to the target".

| improve this answer | |
$\endgroup$
  • $\begingroup$ Ok, so, the target-value network is used to train the action-value network. And vice versa? $\endgroup$ – Alter Jul 21 '19 at 19:13
  • $\begingroup$ @Alter: You didn't ask about the networks, but the values which is different (you may not of realised that of course) . . . yes to first part in basic DQN, no to second part. Instead the target network is updated every so many steps (a few thousand) by simply copying the parameters. However in double learning there are more networks and they do alternate roles between generating target value and being updated. Stick to your understanding of target value from this answer though, and this will all make sense . . . $\endgroup$ – Neil Slater Jul 21 '19 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.