Temporal difference learning with a neural network

Suppose I would like to train a value network $$v$$ via TD(0).

So my TD target for a time step $$t$$ equals:

$$R_{t+1} + \gamma v(s_{t+1})$$

If I understand correctly, I just need to use mean squared error, so that $$v(s_t)$$ becomes closer to this target. However, my network outputs values between $$(-1; 1)$$ and rewards are from this interval also, so the TD target lies between $$(-2; 2)$$. Should I scale it before apply learning? What are the consequences of not doing this i.e. training a neural network with target values from a broader interval that it's output? Can we say anything about it from theoretical point of view?

Should I scale it before apply learning?

Definitely, you need to either:

1. Scale your network output so that it lies in the range (-2, 2), or,
2. Scale your training data so that it lies in the range (-1, 1)

The latter is marginally preferable because it will only need to be computed once, rather than on every pass through your network.

Can we say anything about it from theoretical point of view?

This is a very standard design, and is simply a consequence of the fact that most commonly chosen activation functions have bounds of [-1, 1], [0, 1], [0, inf) or (-inf, inf). In particular for finite bounded activation functions (i.e. those with outputs bounded by [-1, 1] or [0, 1]), you will need to make sure that the output space of the network is capable of representing the desired target space. For example, for binary classification problems, you will often see people choosing labels of either 0/1 or -1/1 depending on whether the final output is activated by a tanh or sigmoid function.

• Ok, but can you say what are the implications of running training (backpropagation) of a neural network with output in [-1, 1] on data that is from [-2; 2]? Does this approach have to fail? – xan Jul 10 '19 at 11:47
• It will probably fail. You will have massive errors for all your samples which do not lie in the interval [-1, 1], which will push your network to overcompensate to try and minimise them. In the best case, you will have very unstable convergence; in the worst case, the training will collapse completely. Really there is no reason to not scale your training data to match the output of your network. – timchap Jul 10 '19 at 11:56