11
$\begingroup$

Feature scaling can be crucially necessary when using distance-, variance- or gradient-based methods (KNN, PCA, neural networks...), because depending on the case, it can improve the quality of results or the computational effort.

In some cases (tree-based models in particular), scaling has no impact on the performance.

There are many discussions out there about when one should scale their features, and why they should do it. Apart from interpretability (which is not a problem as long as the scaling can be reverted), I'm wondering about the opposite: are there cases when scaling is a bad idea, i.e. can have a negative impact on model quality? or less importantly, on computation time?

Improve this question
$\endgroup$
2
  • $\begingroup$ Is this general question or you're facing some particular problem? $\endgroup$
    – Piotr Rarus
    Commented Dec 6, 2019 at 12:26
  • $\begingroup$ General question. I wonder if performing systematic scaling might do something nasty that I would miss. $\endgroup$
    – Romain Reboulleau
    Commented Dec 6, 2019 at 22:40

6 Answers 6

Reset to default
7
+50
$\begingroup$

Scaling often assumes you know the min/max or mean/standard deviation, so directly scaling features where these information is not really known, can be a bad idea.

For example, clipped signals may hide this info, so scaling them can have a negative result because you may distort its true values.

Below is an image of 1) a signal that can be scaled, and 2) a clipped signal that scaling should not be done.

https://mackie.com/blog/what-clipping

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for your answer. So if I develop, a case where I should not scale is when I have reasons to believe that the dataset does not well represent the distribution metrics used for scaling: that makes sense. I think that standardization is more robust to this kind of issue than min-max scaling (especially regarding outliers). $\endgroup$
    – Romain Reboulleau
    Commented Dec 9, 2019 at 17:55
  • $\begingroup$ Maybe standardization is more robust. Specially with the example I put in my answer. But it is difficult to say because we don't know the degree of clipping. Also, you are also kinda assuming a normal distribution. If your data is sampled not from a normal distribution you are also modelling your scaling in the wrong way. $\endgroup$
    – Bruno Lubascher
    Commented Dec 10, 2019 at 17:48
1
$\begingroup$

The example that comes to mind is images ; I’ve never heard of scaling pixel intensities before processing with CNN. Presumably it’s useful to maintain mean differences between the features — eg it could be signal that the top right corner is usually less red , etc .

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ There are some image preprocessing techniques that I would consider scaling (e.g. histogram equalization) that can improve image quality. $\endgroup$
    – Bruno Lubascher
    Commented Dec 9, 2019 at 8:36
  • $\begingroup$ I saw many cases that pixels intensities are scaled to [-1, 1] range. $\endgroup$
    – tkarahan
    Commented Dec 9, 2019 at 8:54
  • 1
    $\begingroup$ You almost always scale pixel intensities. After all, a CNN is still a neural network. Have a look at the respective keras module. If you try to use one of the pre-built keras models without scaling the pixel intensities, you will end up with a horrible performance. $\endgroup$
    – georg-dev
    Commented Dec 9, 2019 at 9:16
  • $\begingroup$ Pixels must always be scaled, either in the [0, 1] or in the [-1, 1] interval. These are the regions where activation functions can capture non-linearities $\endgroup$
    – Leevo
    Commented Dec 10, 2019 at 8:50
1
$\begingroup$

If the features are correlated, don't scale them. You can damage your data applying scaling to each feature separately. It depends on your data, problem and operator you'll be aplying.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Can you develop your answer? If two correlated features have highly different scales, not scaling either will certainly lead to bad performance of distance-based approaches, or am I missing something? $\endgroup$
    – Romain Reboulleau
    Commented Dec 7, 2019 at 10:45
  • $\begingroup$ Take images for example. If you scale every pixel separately among your data set, you'll damage it. Only reasonable thing to do, would be to scale whole image, independently from the others. $\endgroup$
    – Piotr Rarus
    Commented Dec 9, 2019 at 7:22
  • $\begingroup$ How would you even scale every pixel separately? You always scale with respect to something. If you scale an image, you scale every pixel with respect to the mean or the min/max-values of the whole image. $\endgroup$
    – georg-dev
    Commented Dec 9, 2019 at 10:05
  • $\begingroup$ Traditionally, you'd scale pixel, with same coordinates, in every image. This is how scalers in sklearn work. You take attribute value from each record and scale it. $\endgroup$
    – Piotr Rarus
    Commented Dec 9, 2019 at 10:16
  • $\begingroup$ I understand your point. If you have many features with possibly different distributions, but representing the same concept (pixel intensity in the case of images), those features should not necessarily be scaled independently from each other. If they are to be scaled, it's better to use a single transformation. In the case where those are the only features, it is thus not even interesting to scale. This specific image example can actually be extended to whatever case involves features representing a single concept (for instance, $(x, y, z)$ position within space). $\endgroup$
    – Romain Reboulleau
    Commented Dec 9, 2019 at 17:50
1
$\begingroup$

An immediate example is standard scaling or whitening data before a PCA. By normalizing each variance, these scalings erase the relative magnitude of the eigenvalues of the covariance matrix. Hence it defeats the purpose of a PCA.

Improve this answer
$\endgroup$
1
$\begingroup$

The majority of features, especially in physical sciences, have names, definitions, values and units (s, m, kg, etc.), not only names and values. Knowing this, it is easy to manually or even automatically create new features basing on the units. It makes no sense to add meters to seconds, but (x1^2+x2^2+x3^2)^0.5, where x1, x2, x3 are the space coordinates of the same unit is potentially a very valuable feature (distance). Scaling before the ~creative feature engineering stage successfully destroys these (hidden for many) dataset properties and decreases the chance to find new valuable features.

Improve this answer
$\endgroup$
0
$\begingroup$

In a regression problem and based on algorithm of your choice (such as multiple linear regression, or symbolic regression) you don't need to scale your data. As I examined in several problems, scaling hurts the model fit when data is scaled. However for SVM ans ANN you may need to scale your data

There is another case that one needs to select right scaling method, which is a dataset with both categorical and numerical variables.

If one uses min/max method, model may be confused to determine that 1 is for numerical or categorical feature (discrete/continuous); especially if one wants to do clustering! So the right method may be standardizing (I am working on such a problem now)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.