0
$\begingroup$

I'm training my CNN network with one model's data whereas i'm testing it with another model's data. I perform min max normalization on each sample. And every samples ranges [-1,1]. My question is that while performing min-max normalization we pick minimum and maximum value in each sample's features. For the test data should I choose minimum and maximum of each sample Or should I normalize it with the minimum and maximum value of training data?

$\endgroup$
0
$\begingroup$

You should use your training data fit function (by fit function i mean the function of the scaling that is learning from your data)simply because your test dataset size and feature values could be different than your training which would result in a scale different than the scale used on your training data. imagine you want to infer your model with 1 row which will be a problem if you try to fit 1 row on the minmaxscaler.

Example : Let's say your max value is 100 and min value is 0 on your training data for a particular feature, you scale that to [0,1] => 100 will be 1, 0 will stay as so. 70 will be 0.7 , 60 = 0.6 and so on. You move on to the test set and apply scaling on it but it happens that the max value for that same feature in your test set is 70 ( 100 unfortunately for you happens to exist only in the training set ). Your 70 in the test set if you scale using the test set will be a 1. You feed the 1 ( 70 ) to your model who's trained to consider a 1 as a 100 thus the error.

$\endgroup$
  • $\begingroup$ what do you mean by training data fit function? I have 10000 training samples, each samples has 256 features and I normalize each sample 's feature range [-1,1]. In order to perform [-1,1] normalization, I have to pick each minimum and maximum value in each sample's 256 features. But my test data has only 700 samples. I'm little confused that how can I pick the scale of 10000 samples to normalize 700 samples. $\endgroup$ – jerry Jun 20 at 16:17
  • $\begingroup$ i'll edit my answer for you to understand better. $\endgroup$ – Blenz Jun 20 at 16:22
  • $\begingroup$ Actually I have total 256 features in one sample (1 row). I am noemalizing these 256 features such that 1st feature is -1 and last feature is 256. That is why i'm confusing while implementing it on test data. I'm not picking the maximum and minimum value of some particular feature. Can you please explain how should I normalize in this case? $\endgroup$ – jerry Jun 20 at 17:02
  • $\begingroup$ I actually understand what you explained. But my normalization is different from what you explained $\endgroup$ – jerry Jun 20 at 17:05
  • $\begingroup$ the explanation applies for any transformation(that learns from the data) you apply to your training/test datasets only for one exception when you are sure that your training/test datasets are similar in range $\endgroup$ – Blenz Jun 20 at 17:13
0
$\begingroup$

You should always normalize the test data with the parameters/techniques used for training data.

Example from sklearn link,

Here, scaler.fit() on the data learns the parameters and normalizes the data using transform. The same parameters are used to transform the data [2, 2] (test data). The test data size here is irrelevant as the learnt parameters (from training data) are used to convert each of the test samples.

>>> from sklearn.preprocessing import MinMaxScaler
>>> data = [[-1, 2], [-0.5, 6], [0, 10], [1, 18]]
>>> scaler = MinMaxScaler()
>>> print(scaler.fit(data))
MinMaxScaler(copy=True, feature_range=(0, 1))
>>> print(scaler.data_max_)
[ 1. 18.]
>>> print(scaler.transform(data))
[[0.   0.  ]
 [0.25 0.25]
 [0.5  0.5 ]
 [1.   1.  ]]
>>> print(scaler.transform([[2, 2]]))
[[1.5 0. ]]
$\endgroup$
  • $\begingroup$ you normalized each feature. My normalization is different. My normalization will generate [[0,1],[0,1],[0,1],[0,1]] instead of assigning 0 each feature's minimum as zero and maximum as 1 $\endgroup$ – jerry Jun 20 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.