11
$\begingroup$

Feature scaling can be crucially necessary when using distance-, variance- or gradient-based methods (KNN, PCA, neural networks...), because depending on the case, it can improve the quality of results or the computational effort.

In some cases (tree-based models in particular), scaling has no impact on the performance.

There are many discussions out there about when one should scale their features, and why they should do it. Apart from interpretability (which is not a problem as long as the scaling can be reverted), I'm wondering about the opposite: are there cases when scaling is a bad idea, i.e. can have a negative impact on model quality? or less importantly, on computation time?

$\endgroup$
  • $\begingroup$ Is this general question or you're facing some particular problem? $\endgroup$ – Piotr Rarus - Reinstate Monica Dec 6 '19 at 12:26
  • $\begingroup$ General question. I wonder if performing systematic scaling might do something nasty that I would miss. $\endgroup$ – Romain Reboulleau Dec 6 '19 at 22:40
7
+50
$\begingroup$

Scaling often assumes you know the min/max or mean/standard deviation, so directly scaling features where these information is not really known, can be a bad idea.

For example, clipped signals may hide this info, so scaling them can have a negative result because you may distort its true values.

Below is an image of 1) a signal that can be scaled, and 2) a clipped signal that scaling should not be done.

https://mackie.com/blog/what-clipping

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your answer. So if I develop, a case where I should not scale is when I have reasons to believe that the dataset does not well represent the distribution metrics used for scaling: that makes sense. I think that standardization is more robust to this kind of issue than min-max scaling (especially regarding outliers). $\endgroup$ – Romain Reboulleau Dec 9 '19 at 17:55
  • $\begingroup$ Maybe standardization is more robust. Specially with the example I put in my answer. But it is difficult to say because we don't know the degree of clipping. Also, you are also kinda assuming a normal distribution. If your data is sampled not from a normal distribution you are also modelling your scaling in the wrong way. $\endgroup$ – Bruno Lubascher Dec 10 '19 at 17:48
1
$\begingroup$

The example that comes to mind is images ; I’ve never heard of scaling pixel intensities before processing with CNN. Presumably it’s useful to maintain mean differences between the features — eg it could be signal that the top right corner is usually less red , etc .

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ There are some image preprocessing techniques that I would consider scaling (e.g. histogram equalization) that can improve image quality. $\endgroup$ – Bruno Lubascher Dec 9 '19 at 8:36
  • $\begingroup$ I saw many cases that pixels intensities are scaled to [-1, 1] range. $\endgroup$ – Tolga Karahan Dec 9 '19 at 8:54
  • 1
    $\begingroup$ You almost always scale pixel intensities. After all, a CNN is still a neural network. Have a look at the respective keras module. If you try to use one of the pre-built keras models without scaling the pixel intensities, you will end up with a horrible performance. $\endgroup$ – georg-un Dec 9 '19 at 9:16
  • $\begingroup$ Pixels must always be scaled, either in the [0, 1] or in the [-1, 1] interval. These are the regions where activation functions can capture non-linearities $\endgroup$ – Leevo Dec 10 '19 at 8:50
1
$\begingroup$

If the features are correlated, don't scale them. You can damage your data applying scaling to each feature separately. It depends on your data, problem and operator you'll be aplying.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Can you develop your answer? If two correlated features have highly different scales, not scaling either will certainly lead to bad performance of distance-based approaches, or am I missing something? $\endgroup$ – Romain Reboulleau Dec 7 '19 at 10:45
  • $\begingroup$ Take images for example. If you scale every pixel separately among your data set, you'll damage it. Only reasonable thing to do, would be to scale whole image, independently from the others. $\endgroup$ – Piotr Rarus - Reinstate Monica Dec 9 '19 at 7:22
  • $\begingroup$ How would you even scale every pixel separately? You always scale with respect to something. If you scale an image, you scale every pixel with respect to the mean or the min/max-values of the whole image. $\endgroup$ – georg-un Dec 9 '19 at 10:05
  • $\begingroup$ Traditionally, you'd scale pixel, with same coordinates, in every image. This is how scalers in sklearn work. You take attribute value from each record and scale it. $\endgroup$ – Piotr Rarus - Reinstate Monica Dec 9 '19 at 10:16
  • $\begingroup$ I understand your point. If you have many features with possibly different distributions, but representing the same concept (pixel intensity in the case of images), those features should not necessarily be scaled independently from each other. If they are to be scaled, it's better to use a single transformation. In the case where those are the only features, it is thus not even interesting to scale. This specific image example can actually be extended to whatever case involves features representing a single concept (for instance, $(x, y, z)$ position within space). $\endgroup$ – Romain Reboulleau Dec 9 '19 at 17:50
1
$\begingroup$

An immediate example is standard scaling or whitening data before a PCA. By normalizing each variance, these scalings erase the relative magnitude of the eigenvalues of the covariance matrix. Hence it defeats the purpose of a PCA.

| improve this answer | |
$\endgroup$
1
$\begingroup$

The majority of features, especially in physical sciences, have names, definitions, values and units (s, m, kg, etc.), not only names and values. Knowing this, it is easy to manually or even automatically create new features basing on the units. It makes no sense to add meters to seconds, but (x1^2+x2^2+x3^2)^0.5, where x1, x2, x3 are the space coordinates of the same unit is potentially a very valuable feature (distance). Scaling before the ~creative feature engineering stage successfully destroys these (hidden for many) dataset properties and decreases the chance to find new valuable features.

| improve this answer | |
$\endgroup$
0
$\begingroup$

In a regression problem and based on algorithm of your choice (such as multiple linear regression, or symbolic regression) you don't need to scale your data. As I examined in several problems, scaling hurts the model fit when data is scaled. However for SVM ans ANN you may need to scale your data

There is another case that one needs to select right scaling method, which is a dataset with both categorical and numerical variables.

If one uses min/max method, model may be confused to determine that 1 is for numerical or categorical feature (discrete/continuous); especially if one wants to do clustering! So the right method may be standardizing (I am working on such a problem now)

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.