# Tag Info

40

The most logical way to transform hour is into two variables that swing back and forth out of sink. Imagine the position of the end of the hour hand of a 24-hour clock. The x position swings back and forth out of sink with the y position. For a 24-hour clock you can accomplish this with x=sin(2pi*hour/24),y=cos(2pi*hour/24). You need both variables or the ...

27

Lat long coordinates have a problem that they are 2 features that represent a three dimensional space. This means that the long coordinate goes all around, which means the two most extreme values are actually very close together. I've dealt with this problem a few times and what I do in this case is map them to x, y and z coordinates. This means close points ...

25

This is called unity-based normalization. If you have a vector $X$, you can obtain a normalized version of it, say $Z$, by doing: $$Z = \frac{X - \min(X)}{\max(X) - \min(X)}$$

21

We love the normal form In most cases we try to make them act like normal. Its not classifiers point of view but its feature extraction view! Which Transformation? The main criterion in choosing a transformation is: what works with the data? As above examples indicate, it is important to consider as well two questions. What makes physical (biological, ...

19

You might want to interpret your coefficients. That is, to be able to say things like "if I increase my variable $X_1$ by 1, then, on average and all else being equal, $Y$ should increase by $\beta_1$". For your coefficients to be interpretable, linear regression assumes a bunch of things. One of these things is no multicollinearity. That is, your $X$ ...

16

Once converted to numerical form, models don't respond differently to columns of one-hot-encoded than they do to any other numerical data. So there is a clear precedent to normalise the {0,1} values if you are doing it for any reason to prepare other columns. The effect of doing so will depend on the model class, and type of normalisation you apply, but I ...

15

I once heard a data scinetist state at a conference talk: "Basically, you can do what you want, as long as you know what you are doing." This also applies here. The more statistically sound way would be to transform all variables prior to additional steps such as PCA or factor analysis. Then you still know the scale of your variables and can interpret the ...

9

The questions of whether and why it's important, depends on the context. For gradient boosted decision trees, for example, it is not important - these ML algorithms "don't care" about monotone transformations to the data; they just look for points to split it. For linear predictors, for example, scaling can improve interpretability of the results. If you'd ...

8

You cannot really use k-means clustering if your data contains categorical variables since k-means uses Euclidian distance which will not make a lot of sense with categorical variables. Check out the answers to this similar question. You can use the following rules for performing clustering with k-means or one of its derivates: If your data contains only ...

7

This was meant as a comment but it is too long. The fact that your test set has a different range might be a sign that the training set is not a good representation of the test set. However, if the difference is really small as in your example, it is likely that it won't affect your predictions. Unfortunately, I don't think I have a good reason to think it ...

7

Scaling often assumes you know the min/max or mean/standard deviation, so directly scaling features where these information is not really known, can be a bad idea. For example, clipped signals may hide this info, so scaling them can have a negative result because you may distort its true values. Below is an image of 1) a signal that can be scaled, and 2) a ...

6

Within each class, you'll have distributions of values for the features. That in itself is not a reason for concern. From a slightly theoretical point of view, you can ask yourself why you should scale your features and why you should scale them in exactly the chosen way. One reason may be that your particular training algorithm is known to converge faster ...

6

A detailed answer to the question can be found here. [...]are there times when it is not appropriate or not beneficial? Short answer: Yes and No. Yes in the terms, that it can significantly change your output of e.g. clustering algorithms. No, on the other hand, if these changes are what you want to achieve. Or to put it in the words of the author of ...

5

First you do not always need to normalize (standardize) the input vectors (feature vectors), sometimes is good, sometimes is bad. In general you scale your feature vector when the magnitude of a feature dominates the others, so the model cannot pick up the contribution of the smaller magnitude features. Read here for a detailed explanation. Second there ...

5

Clustering algorithms are certainly effected by the feature scaling. Example: Let's say that you have two features: weight (in Lbs) height (in Feet) ... and we are using these to predict whether a person needs a 'S' or 'L' size shirt. We are using weight+height for that, and in our trained set let's say we have two people already in clusters: Adam (...

5

You should apply the normalization only on your training dataset. Your test set should be kept completely separate and should be used only when your final model has been chosen. If you use include the testing set in the normalization, it can be seen as using the testing set in the training procedure. This is called data snooping. You should pre-process ...

5

You can not use PCA, or at least it is not recommended, for mixed data. It is best to use Factor analysis of mixed data. You are lucky that Prince is a Python package that covers all data scenarios, borrowing from its explanation: All your variables are numeric: use principal component analysis (prince.PCA) You have a contingency table: use ...

5

The most accepted idea is that bag-of-words, Tf-Idf and other transformations should be left as is. According to some: Standardization of categorical variables might be not natural. Neither is standarization of Tf-Idf because according to stats stack exchange: (it's) (...) usually is a two-fold normalization. First, each document is normalized to ...

4

The short answer is that the y_original and x_reduced are still connected to each other, so it is safe to train your data using y_original and x_reduced. While x_reduced is on a different scale, as you mentioned via eigenvectors, it still is representative of the data that was attached to that observation, just in a different format. You lose a lot of ...

4

Yes. Clustering algorithms such as K-means do need feature scaling before they are fed to the algo. Since, clustering techniques use Euclidean Distance to form the cohorts, it will be wise e.g to scale the variables having heights in meters and weights in KGs before calculating the distance.

4

The fact that the coefficients of hp and disp are low when data is unscaled and high when data are scaled means that these variables help explaining the dependent variable but their magnitude is large, so the coefficients in the unscaled case have to be low. In terms of "importance", I would say that the absolute value of the coefficients in the scaled case ...

4

You can't really talk about significance in this case without standard errors; they scale with the variables and coefficients. Further, each coefficient is conditional on the other variables in the model, and collinearity actually seems to be inflating the importance of hp and disp. Rescaling variables should not change the significance of results at all. ...

4

You are refitting scaler_x on your test set, which you don't want. Change this line: xaa = scaler_x.fit_transform(xaa) to xaa = scaler_x.transform(xaa) You are getting [-1, -1, ..., -1] because with one sample, each feature is equal to the minimum.

4

StandardScaler and MinMaxScaler are more common when dealing with continuous numerical data. One possible preprocessing approach for OneHotEncoding scaling is "soft-binarizing" the dummy variables by converting softb(0) = 0.1, softb(1) = 0.9. From my experience with feedforward Neural Networks this was found to be quite useful, so I expect it to ...

4

I suggest to try a log transformation. This has two potential benefits: The range of x values becomes smaller Your transformed data might be closer to resemble a normal distribution (only relevant for some models, e.g. not for trees) Here are two toy examples to illustrate: Toy example 1 s = np.random.lognormal(3, 1, 1000) plt.hist(s, 100) plt.show() ...

4

It affects anything optimized by a form of gradient descent, because it affects the relative scale of the dimensions of the input. If A is generally 1000x larger than B, then changing B's coefficient by some amount is in a sense a 1000x bigger move. In theory this won't matter but in practice it can cause the gradient descent to have trouble landing in the ...

4

Scaling doesn't affect the performance of any tree-based method, not for lightgbm, xgboost, catboost or even a decision tree. This post that elaborates on the topic, but mainly the issue is that decision trees split the feature space based on binary decisions like "is this feature bigger than this value?", and if you scale your data, the decisions ...

4

You're also scaling $y$, then of course you are getting lower error. That question was regarding scaling $X$. The same model will have very different error metrics when units on $y$ are changed: if I multiply all $y$ values by 100, the error will be 100 times larger, if I divide all $y$ values by 100 the error will be divided by 100.

3

The skewed data here is being normalised by adding one(one added so that the zeros are being transformed to one as log of 0 is not defined) and taking natural log. The data can be nearly normalised using the transformation techniques like taking square root or reciprocal or logarithm. Now, why it is required. Actually many of the algorithms in data assume ...

3

Find the largest positive number and the smallest (most negative) number in the array. Add the absolute value of the smallest (most negative) number to every value in the array. Divide each result by the difference between the largest and the smallest number.

Only top voted, non community-wiki answers of a minimum length are eligible