3
$\begingroup$

I wrote a Neural Networks prediction model in Python.

My data has a few inputs and two outputs. In order to make it work, I have to normalise every column on data for good prediction results.

However, I had an issue. I run several times the prediction model with the same inputs to get the average and standard deviation of it. But obviously, those are also normalised to 0-1. On the test data I knew the min and max, so I could denormalised them. On the prediction values, I cannot the min and max real value.

How do you solve this kind of problem and if you cannot, are there any other decent prediction techniques without the need of normalisation?

$\endgroup$
2
$\begingroup$

Normalise using your training data statistics. Save the values used (e.g. mean and sd per feature), treating them as part of your model. Once you have used these values to transform input, they become fixed translate/scale factors in the model.

Use the same values to normalise test data or new inputs as required. You do not need to calculate new normalisation constants for new data. In fact doing so will most likely reduce the effectiveness of your model.

The same principle applies to interpreting output values if you need to scale those into range that your model produces. Scale according to your training data.

$\endgroup$
1
$\begingroup$

Why not use the same statistics to denormalize the results? If the test data is good enough to train a neural network then why would its statistics not be good enough to denormalize the results?

However, I would suggest not using min and max as the scale factors. The min and max can be very sensitive. I suggest using robust estimates of the mean and standard deviation to normalize.

$\endgroup$
  • $\begingroup$ The max and min of the training dataset are not necessary the same for the predictive dataset. Neither the average and the ST. Dev. So, the assumption to use the same statistics is not right. $\endgroup$ – Tasos Jul 3 '15 at 13:51
  • 1
    $\begingroup$ Like I said, I agree that the min and max are not good scale factors. They are too sensitive to outliers. However, if the robust statistics of mean and standard deviation the two datasets are that much different than why are you training with that dataset? It would seem that you should not be using a training set that is so far away from the prediction set. $\endgroup$ – dpmcmlxxvi Jul 3 '15 at 17:31
  • $\begingroup$ This is good advice... @dpmcmlxxvi is right that 1) standard deviation tends to be more robust than min and max and 2) the validity of your model depends on the mean and standard deviation of your training data set being similar to the mean and standard deviation of your test data set. $\endgroup$ – AN6U5 Jul 31 '15 at 2:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.