3
$\begingroup$

For instance, many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the l1 and l2 regularizers of linear models) assume that all features are centered around zero and have variance in the same order.

This is from the scikit-learn

Can someone please specify what elements they are referring when it is said the mean is expected to be zero.

I understand that the variance should be in similar range for the algorithms to give same significance to each feature. But is there anything also necessarily expecting zero mean in the data?

In other words, if I know variance of each feature is already in same range, can something still go wrong as it says some "elements" expect zero mean.

$\endgroup$
1
$\begingroup$

The reason is the same.
I assume you understand how the Features at a very different scale can create issue

But just scaling will not always bring them on the similar scale because Standard Deviation is dependent on the Range of the Feature.
So, if a feature is very large but in a small range then simply scaling it will not help.

Let's check an example with two features

import numpy as np
np.set_printoptions(precision=2)

feat_A = np.array([1,2,3,4,5])
feat_B = np.array([10000000,10000001,10000002,10000003,10000005])

# Case I  - Only Std
print(feat_A/feat_A.std())
print(feat_B/feat_B.std())

# Case II - Mean and Std
print((feat_A-feat_A.mean())/feat_A.std())
print((feat_B-feat_B.mean())/feat_B.std())

Output

Case - I - [You may see, this doesn't solve the problem]

[0.71 1.41 2.12 2.83 3.54]
[5812381.94 5812382.52 5812383.1 5812383.68 5812384.84]

Case - II - [This is what we were looking for]

[-1.41 -0.71 0. 0.71 1.41]
[-1.28 -0.7 -0.12 0.46 1.63]

$\endgroup$
1
  • $\begingroup$ Hi, Thank you very much for an answer. I know just dividing by variance is not a standardization of data. What I am looking for is a more clear wording in the documentation compared to what is stated presently. I just want to make sure that having zero mean is not a necessity if all features are already in the same range. $\endgroup$ – Shivam Arora Feb 6 at 20:16
1
$\begingroup$

It seems very unlikely that centering would hurt, and so I'd suggest just to do it anyway.

Theoretically, in a generalized linear model with regularization, no, centering won't change anything. This is because the intercept term can absorb any changes; shifting $x$ by 100 can simply be rewritten: $$ 15 + 0.2\cdot(x-100) = 15 - 0.2\cdot100 + 0.2x = -5 + 0.2x,$$ so that essentially the same model exists for the shifted data with the same coefficient penalty. Of course, that suggests the first nontrivial example where failing to center will hurt performance: if you don't fit an intercept term!

However, in trying to test that in sklearn, I have some troubles (notebook): scaling with and without centering gives different results in penalized logistic regression, depending on the solver! (saga gives different results, but lbfgs gives nearly-identical coefficients.) I'm not sure yet if this is some numerical issue (which I have seen before, but with datasets of much more varying scales) or a misunderstanding on my part. Penalized linear regression seems to work fine.

Finally, to SVMs with the rbf kernel. My understanding of nonlinear-kernel SVMs is rather limited, but the rbf kernel should be invariant to centering. However, see Data Centering in Feature Space (pdf) which posits that centering in the kernel space can be helpful. Whether for theoretical or numerical reasons, both the classification and regression versions in the above-linked notebook show different results after centering.

$\endgroup$
4
  • $\begingroup$ Hi, thanks for an answer and the notebook. I think just_scale in your notebook StandardScaler(with_mean = False) will do exactly what 10xAI wrote in the answer. So it make sense that it would not perform same as doing proper standardization, where each feature would be in the same range (-1,1) numerically. $\endgroup$ – Shivam Arora Feb 6 at 23:36
  • 1
    $\begingroup$ The point is that with_mean=False doesn't center, but does scale to unit variance. 10xAI's answer addresses what the difference in the data looks like, but the main question is how/whether/when this affects the model. $\endgroup$ – Ben Reiniger Feb 6 at 23:40
  • $\begingroup$ Thanks for the sharing the paper on centering the data. It sounds interesting. That was sort of the motivation; if such methods are embedded in the sklearn. I am playing with load_digits and it does not seem reasonable to me to standardize it as they are all in same range already. Also, I didn't see any comment anywhere on the loss of precision and increase in computation time dealing with transformed floating data. I did see some papers using less precision float to make things faster but nothing on avoiding such transformations wherever you can. Could you please say something on that? $\endgroup$ – Shivam Arora Feb 6 at 23:59
  • $\begingroup$ Yes you are right, I haven't seen any explanation on that. The problem of 'difference in the distance magnitude' is already handled by scaling the variance. I guess one possible place where it might be getting affected might be to do with some of gradient decent algorithms and learning rate sizes? $\endgroup$ – Shivam Arora Feb 7 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.