I'm trying to understand the difference between target-values and action-values in Deep Q Networks.
From what I understand, action-value tries to approximate the reward of a given action (at some state). The target-value is also an approximate of the reward. How are they different?
Reference
- https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf
Action value and target value in DQN refer to the same thing in terms of what they measure, but are obtained and used in different ways.
An action value does not approximate the reward of a given action, but represents the related concept of expected return - the expected discounted sum of future rewards when taking action $a$ in state $s$. You see it very often associated with the action value function $q(s,a)$.
Action values are not inherently approximated, they are more conceptual than that. However, in practice in Reinforcement Learning (RL), the action value function must be learned, and some form of approximation used based on observed data. When an action value function it is approximated by a neural network during learning you may see it written $\hat{q}(s,a,\theta)$ which makes explicit that the NN is approximating some "true" function that you don't know, and that it is parametrised by $\theta$, the neural network weights and biases.
During learning, the agent takes actions and observes resulting states and rewards. It must use these observations to improve its estimates of values. To do so in DQN, the agent constructs a temporal difference (TD) target - for single-step Q-learning this is $G_{t:t+1} = r_{t+1} + \gamma\text{max}_{a'}\hat{q}(s_{t+1},a',\theta)$. This is the "target value" and there are multiple ways to construct it, giving rise to variants of RL algorithms.
The value $G_{t:t+1}$ is also an estimate of expected return. Technically it is a sampled estimate, which may vary and may be biased (due to starting conditions of the neural network). However, it includes some real experience, so may be used to improve the neural network through training. You can use it like ground truth associated with the state and action, to train the neural network.
The TD target or "target value" gets its name because by updating a Q table or training a NN with it as a ground truth, the estimator will output values in future closer to the supplied value. The estimator "gets closer to the target".
