Timeline for Should we normalize test data by choosing maximum and minimum value of training data?
Current License: CC BY-SA 4.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 20, 2019 at 17:37 | comment | added | Blenz | Let us continue this discussion in chat. | |
| Jun 20, 2019 at 17:34 | comment | added | Blenz | No, don't normalize using only 50 samples from the training set, take the max-min values from the whole 1000 samples to create the scale! once you have the scale, you can apply it to your 50 samples and get your normalized test data! | |
| Jun 20, 2019 at 17:32 | comment | added | jerry | I completely understand your question but my question is little different. If I have 1000 training samples at temperature T1 in one model and 50 samples(as test set) at T1 from second model. can I choose any 50 samples from the training data to normalize 50 samples of test data? | |
| Jun 20, 2019 at 17:26 | comment | added | Blenz | Your model ( trained on training data ) should be fed values following one scale ( learned from your training data ), if , for each test dataset, you want to create a new scale like i explained in my answers, you'll have , after scaling , many 1's fed to the model while they don't represent the max value you trained your model on, but the max value available in your test dataset which will induce errors in predictions. | |
| Jun 20, 2019 at 17:23 | comment | added | jerry | My training and test data have different ranges but then how to choose standard value to normalize test data because it has only 500 samples and training set has 10000 samples. | |
| Jun 20, 2019 at 17:13 | comment | added | Blenz | the explanation applies for any transformation(that learns from the data) you apply to your training/test datasets only for one exception when you are sure that your training/test datasets are similar in range | |
| Jun 20, 2019 at 17:05 | comment | added | jerry | I actually understand what you explained. But my normalization is different from what you explained | |
| Jun 20, 2019 at 17:02 | comment | added | jerry | Actually I have total 256 features in one sample (1 row). I am noemalizing these 256 features such that 1st feature is -1 and last feature is 256. That is why i'm confusing while implementing it on test data. I'm not picking the maximum and minimum value of some particular feature. Can you please explain how should I normalize in this case? | |
| Jun 20, 2019 at 16:31 | history | edited | Blenz | CC BY-SA 4.0 |
added 580 characters in body
|
| Jun 20, 2019 at 16:25 | history | edited | Blenz | CC BY-SA 4.0 |
added 580 characters in body
|
| Jun 20, 2019 at 16:22 | comment | added | Blenz | i'll edit my answer for you to understand better. | |
| Jun 20, 2019 at 16:17 | comment | added | jerry | what do you mean by training data fit function? I have 10000 training samples, each samples has 256 features and I normalize each sample 's feature range [-1,1]. In order to perform [-1,1] normalization, I have to pick each minimum and maximum value in each sample's 256 features. But my test data has only 700 samples. I'm little confused that how can I pick the scale of 10000 samples to normalize 700 samples. | |
| Jun 20, 2019 at 16:11 | history | answered | Blenz | CC BY-SA 4.0 |