2
$\begingroup$

I have following pipeline:

estimators = []
estimators.append(('standardize', StandardScaler()))
prepare_data = Pipeline(estimators)

Originally, the data looks like

data_unstandardscaled

After applying the estimator/StandardScaler()

X_train = prepare_data.fit_transform(X_train_raw)

data_standardscaled

Why are the values mostly/always negative now?

When I have a look at another variable in X_train it looks as it should (I guess):

Before

data_unstandardscaled2

after

data_standardscaled2

$\endgroup$

1 Answer 1

1
$\begingroup$

You got outliers. Standard scaler scales each attribute independently to center at zero and unit variance. Either deal with your outliers or use some more robust scaler. Try instead to plot each attribute with separate box plot.

$\endgroup$
7
  • $\begingroup$ I think I understood it at least partially :) Outliers are the reason why it is far off from +1/-1, right? But shouldn't it be centered, anyways? $\endgroup$
    – Ben
    Nov 22, 2019 at 12:45
  • $\begingroup$ What exactly your plots represent? Mean for each attribute? $\endgroup$ Nov 22, 2019 at 12:48
  • $\begingroup$ @Ben You should check what is the least negative value of your variable. Mean is not robust to outliers, that is for sure, or you can check with histogram, as piotr says box plot helps as well. $\endgroup$ Nov 22, 2019 at 12:56
  • $\begingroup$ @PiotrRarus It is current and gas concentration. $\endgroup$
    – Ben
    Nov 25, 2019 at 6:16
  • 2
    $\begingroup$ @Ben, if you want to understand what's going on you need to get your hands dirty: do fit and store the obtained scaler. Only then call transform on that scaler. Inspect the attributes of the scaler .mean_ and .scale_ (std of x). Then go and visually check some of the scaled values. I suspect most are actually positive but small so we barely notice them on the plot, which then looks bias due to those extreme low peaks -not all are outliers I'd say. However, the values might be correct. Remember, the scaled values are $Z=\frac{x-\bar{x}}{S_x}$ You must have some positive values if done right. $\endgroup$
    – MASL
    Jan 3, 2020 at 6:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.