Temporal difference learning with a neural network

Question

Suppose I would like to train a value network $v$ via TD(0).

So my TD target for a time step $t$ equals:

$$R_{t+1} + \gamma v(s_{t+1})$$

If I understand correctly, I just need to use mean squared error, so that $v(s_t)$ becomes closer to this target. However, my network outputs values between $(-1; 1)$ and rewards are from this interval also, so the TD target lies between $(-2; 2)$. Should I scale it before apply learning? What are the consequences of not doing this i.e. training a neural network with target values from a broader interval that it's output? Can we say anything about it from theoretical point of view?

Answer 1

Should I scale it before apply learning?

Definitely, you need to either:

1. Scale your network output so that it lies in the range (-2, 2), or,
2. Scale your training data so that it lies in the range (-1, 1)

The latter is marginally preferable because it will only need to be computed once, rather than on every pass through your network.

Can we say anything about it from theoretical point of view?

This is a very standard design, and is simply a consequence of the fact that most commonly chosen activation functions have bounds of [-1, 1], [0, 1], [0, inf) or (-inf, inf). In particular for finite bounded activation functions (i.e. those with outputs bounded by [-1, 1] or [0, 1]), you will need to make sure that the output space of the network is capable of representing the desired target space. For example, for binary classification problems, you will often see people choosing labels of either 0/1 or -1/1 depending on whether the final output is activated by a tanh or sigmoid function.