I have standardized training set using

mean = XTrain.mean()

std = XTrain.std()

And then used mean and std to standardize validation and test sets. The training and validation sets have values that are greater than 1 and less than zero is that okay?

  • $\begingroup$ I assume this is just a typo but in line 5 you need to divide by the standard deviation instead of the mean again $\endgroup$ – Sammy Jul 20 '20 at 17:33
  • $\begingroup$ @Sammy yes fixed that $\endgroup$ – skrrrt Jul 20 '20 at 17:53

Standardization centers the values around a mean of $0$ with standard deviation $1$. Therefore, having values smaller than $0$ or greater than $1$ is to be expected. If you want to make sure values are between $0$ and $1$ you need to normalize the data instead.

Here is an example of the two procedures taken from the book "Python Machine Learning" by Raschka:

enter image description here

Be aware though to apply the procedure to your test data with parameters obtained from the training data (in case of standardization: mean and std. dev. of train data).

Sklearn has methods for standardization and normalization which you might want to have a look at.

  • $\begingroup$ Due to some reason my model was performing worse (AUC) when normalized instead of standardized. Is this normal? Also I am performing these operations myself because I have variable length arrays so sklearn's classes don't work there. $\endgroup$ – skrrrt Jul 20 '20 at 17:57
  • 1
    $\begingroup$ @skrrrt performance can be different for norm. and stand. It depends on your data and type of model. See, for example, this question. $\endgroup$ – Sammy Jul 20 '20 at 18:03

You're measuring how many standard deviations from the mean a given value is. Certainly values can be many standard deviations from the mean. Even for data with a normal distribution, we expect about $5\%$ of the observations to be more than $2$ standard deviations from the mean, and we expect $32\%$ of the observations to be more than $1$ standard deviation from the mean.

Therefore, it is not at all concerning that you have values more than $1$.

As far as values less than $0$ go, all that means is that you have a value less than the mean. This is common. (While it can happen, consider how to have a data set where no values are less than the mean.)

As Sammy mentioned mere seconds before I posted, be sure to use the mean and standard deviation from your training data when you transform the test and validation data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.