1
$\begingroup$

I'm doing some preprocessing on my training data before fitting it to a model. Upon checking the results, there is one column that is returning 0 rather than 1 for the standard deviation. (all columns return a mean of 0 as expected). My code is below:

y = ml_df['target']
x = ml_df[['Feature1', 'Feature2',  'Feature3', 'Feature4', 'Feature5', 'Feature6', 
        'Feature7', 'Feature8', 'Feature9', 'Feature10', 'Feature11', 'Feature12']]
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.55, random_state=3)
pt_hp = PowerTransformer()
x_train_gaussian = pt_hp.fit_transform(x_train)
x_test_gaussian =  pt_hp.transform(x_test)

After running the above, this line produces the following output:

print(x_train_gaussian.std(axis = 0))

Out: [1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1.]

This particular feature (histogram below) is not materially different than the others; is contains only positive values (no zeros that would impact PowerTransformer) and not anywhere close to near-zero variance. Does anyone have any idea why this one particular column is returning such a different standard deviation than the rest? Thanks in advance for any suggestions.enter image description here

Edit - added histogram of original feature.

$\endgroup$
3
  • $\begingroup$ It looks like indeed it's the PowerTransformer. I'm not too familiar with these transformers, but (1) what's the value of lambdas_ corresponding to that feature, (2) can you provide a histogram for the original feature, (3) does method='box-cox' suffer the same issue? $\endgroup$
    – Ben Reiniger
    Commented Feb 6, 2022 at 21:55
  • $\begingroup$ Since you say the scaler is redundant, please edit the question title to refer to PowerTransformer instead, and slim down you example to exclude that redundancy. $\endgroup$
    – Ben Reiniger
    Commented Feb 7, 2022 at 0:46
  • $\begingroup$ I've updated the question as suggested. I am getting a fitted lambda of -19.6 for that feature using YeoJohnson; when I try with BoxCox, it's -4.8. No idea why there's such a discrepancy between the two; I would simply fit the Transformer with BoxCox, but several of the other columns have negative values. $\endgroup$
    – AMJ
    Commented Feb 7, 2022 at 3:08

1 Answer 1

1
$\begingroup$

This appears to be a bug, captured in Issue14959, with a linked pull request in the works (slightly stalled now).

The easiest fix for now is just to center your data first.

$\endgroup$
1
  • $\begingroup$ Thanks a lot - would not have guessed it was due to a bug. Will do as suggested in that link and center the data prior to applying the PowerTransformer. $\endgroup$
    – AMJ
    Commented Feb 7, 2022 at 5:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.