Scaling features in artificial neural networks

Question

So it is a well known thing that it is a good idea to scale features/training samples in the training set, so that the values do not differ too much in the absolute sense. For example we want to train a neural network to learn a simple quadratic function y = x*x.

The training data, say, looks like x = [1, 2, 3, 4, 5, 6] for input data and y = [1, 4, 9, 16, 25, 36] for target data.

Obviously, one would scale both the input and the output data by something like x := x/6 and y := y/36, so all the training values live in the [0:1] range.

My question is, what if I then want to predict the output value for the input 7? If I feed 7 / 6 into the network and then rescale the output by doing output := output * 36, I will not get the correct result. How to resolve such an issue?

Answer 1

As per your assumption, if you're scaling the features so that all of them are transformed between 0 and 1, the case where you're inputting 7 will result in a rescaled feature which will be greater than 1 (7/6 > 1). So, this tells us that your scaling technique of dividing inputs by a specific number isn't a really good scaling approach.

For your specific problem, scaling won't really be necessary, as explained here. The important chunk says:

If the input variables are combined linearly, as in an MLP, then it is rarely 
strictly necessary to standardize the inputs, at least in theory. The reason is 
that any rescaling of an input vector can be effectively undone by changing the 
corresponding weights and biases, leaving you with the exact same outputs as you had 
before. However, there are a variety of practical reasons why standardizing the inputs 
can make training faster and reduce the chances of getting stuck in local optima. 
Also, weight decay and Bayesian estimation can be done more conveniently with
standardized inputs. 

But if you're still interested in scaling the data, here's a nice discussion which can be explored to get an idea about standardized scaling/normalization techniques.

Answer 2

Your example is misleading because you only have 1 input feature, therefore the scaling makes no sense. A better example would have at least 2 features. We can write

$$ z = x^2 + y^2 $$

where x and y have values of different orders of magnitude, say x is the age and y the salary and z some characteristic you want to predict. At this point by scaling your model will weight the two features in the same way.