41

The most logical way to transform hour is into two variables that swing back and forth out of sync. Imagine the position of the end of the hour hand of a 24-hour clock. The x position swings back and forth out of sync with the y position. For a 24-hour clock you can accomplish this with x=sin(2pi*hour/24),y=cos(2pi*hour/24). You need both variables or the ...


32

Lat long coordinates have a problem that they are 2 features that represent a three dimensional space. This means that the long coordinate goes all around, which means the two most extreme values are actually very close together. I've dealt with this problem a few times and what I do in this case is map them to x, y and z coordinates. This means close points ...


26

This is called unity-based normalization. If you have a vector $X$, you can obtain a normalized version of it, say $Z$, by doing: $$Z = \frac{X - \min(X)}{\max(X) - \min(X)}$$


21

We love the normal form In most cases we try to make them act like normal. Its not classifiers point of view but its feature extraction view! Which Transformation? The main criterion in choosing a transformation is: what works with the data? As above examples indicate, it is important to consider as well two questions. What makes physical (biological, ...


20

You might want to interpret your coefficients. That is, to be able to say things like "if I increase my variable $X_1$ by 1, then, on average and all else being equal, $Y$ should increase by $\beta_1$". For your coefficients to be interpretable, linear regression assumes a bunch of things. One of these things is no multicollinearity. That is, your $X$ ...


19

Once converted to numerical form, models don't respond differently to columns of one-hot-encoded than they do to any other numerical data. So there is a clear precedent to normalise the {0,1} values if you are doing it for any reason to prepare other columns. The effect of doing so will depend on the model class, and type of normalisation you apply, but I ...


15

I once heard a data scinetist state at a conference talk: "Basically, you can do what you want, as long as you know what you are doing." This also applies here. The more statistically sound way would be to transform all variables prior to additional steps such as PCA or factor analysis. Then you still know the scale of your variables and can interpret the ...


10

The questions of whether and why it's important, depends on the context. For gradient boosted decision trees, for example, it is not important - these ML algorithms "don't care" about monotone transformations to the data; they just look for points to split it. For linear predictors, for example, scaling can improve interpretability of the results. If you'd ...


8

StandardScaler and MinMaxScaler are more common when dealing with continuous numerical data. One possible preprocessing approach for OneHotEncoding scaling is "soft-binarizing" the dummy variables by converting softb(0) = 0.1, softb(1) = 0.9. From my experience with feedforward Neural Networks this was found to be quite useful, so I expect it to ...


8

You cannot really use k-means clustering if your data contains categorical variables since k-means uses Euclidian distance which will not make a lot of sense with categorical variables. Check out the answers to this similar question. You can use the following rules for performing clustering with k-means or one of its derivates: If your data contains only ...


7

This was meant as a comment but it is too long. The fact that your test set has a different range might be a sign that the training set is not a good representation of the test set. However, if the difference is really small as in your example, it is likely that it won't affect your predictions. Unfortunately, I don't think I have a good reason to think it ...


7

The most accepted idea is that bag-of-words, Tf-Idf and other transformations should be left as is. According to some: Standardization of categorical variables might be not natural. Neither is standarization of Tf-Idf because according to stats stack exchange: (it's) (...) usually is a two-fold normalization. First, each document is normalized to ...


7

Scaling often assumes you know the min/max or mean/standard deviation, so directly scaling features where these information is not really known, can be a bad idea. For example, clipped signals may hide this info, so scaling them can have a negative result because you may distort its true values. Below is an image of 1) a signal that can be scaled, and 2) a ...


6

Within each class, you'll have distributions of values for the features. That in itself is not a reason for concern. From a slightly theoretical point of view, you can ask yourself why you should scale your features and why you should scale them in exactly the chosen way. One reason may be that your particular training algorithm is known to converge faster ...


6

A detailed answer to the question can be found here. [...]are there times when it is not appropriate or not beneficial? Short answer: Yes and No. Yes in the terms, that it can significantly change your output of e.g. clustering algorithms. No, on the other hand, if these changes are what you want to achieve. Or to put it in the words of the author of ...


6

You should not use Label Encoding for Categorical data unless there is a known ranking and that also in the specified ratio between the level values. In this case, the model will assume 10 as 2 times of 5. One-hot will add a lot of dimensions as I can see in your data. You must try other Categorical encoding techniques esp. Sum Coding Or Helmert. You ...


5

First you do not always need to normalize (standardize) the input vectors (feature vectors), sometimes is good, sometimes is bad. In general you scale your feature vector when the magnitude of a feature dominates the others, so the model cannot pick up the contribution of the smaller magnitude features. Read here for a detailed explanation. Second there ...


5

Clustering algorithms are certainly effected by the feature scaling. Example: Let's say that you have two features: weight (in Lbs) height (in Feet) ... and we are using these to predict whether a person needs a 'S' or 'L' size shirt. We are using weight+height for that, and in our trained set let's say we have two people already in clusters: Adam (...


5

You should apply the normalization only on your training dataset. Your test set should be kept completely separate and should be used only when your final model has been chosen. If you use include the testing set in the normalization, it can be seen as using the testing set in the training procedure. This is called data snooping. You should pre-process ...


5

You can't really talk about significance in this case without standard errors; they scale with the variables and coefficients. Further, each coefficient is conditional on the other variables in the model, and collinearity actually seems to be inflating the importance of hp and disp. Rescaling variables should not change the significance of results at all. ...


5

You are refitting scaler_x on your test set, which you don't want. Change this line: xaa = scaler_x.fit_transform(xaa) to xaa = scaler_x.transform(xaa) You are getting [-1, -1, ..., -1] because with one sample, each feature is equal to the minimum.


5

You can not use PCA, or at least it is not recommended, for mixed data. It is best to use Factor analysis of mixed data. You are lucky that Prince is a Python package that covers all data scenarios, borrowing from its explanation: All your variables are numeric: use principal component analysis (prince.PCA) You have a contingency table: use ...


5

In "Python Machine Learning" by Raschka the author provides some guidance on page 111 when to normalize (min-max scale) and when to standardize data: Although normalization via min-max scaling is a commonly used technique that is useful when we need values in a bounded interval, standardization can be more practical for many machine learning ...


5

I suggest to try a log transformation. This has two potential benefits: The range of x values becomes smaller Your transformed data might be closer to resemble a normal distribution (only relevant for some models, e.g. not for trees) Here are two toy examples to illustrate: Toy example 1 s = np.random.lognormal(3, 1, 1000) plt.hist(s, 100) plt.show() ...


5

It affects anything optimized by a form of gradient descent, because it affects the relative scale of the dimensions of the input. If A is generally 1000x larger than B, then changing B's coefficient by some amount is in a sense a 1000x bigger move. In theory this won't matter but in practice it can cause the gradient descent to have trouble landing in the ...


5

Scaling doesn't affect the performance of any tree-based method, not for lightgbm, xgboost, catboost or even a decision tree. This post that elaborates on the topic, but mainly the issue is that decision trees split the feature space based on binary decisions like "is this feature bigger than this value?", and if you scale your data, the decisions ...


5

With only 210 samples the difference might be caused by your train and test data not being from the same underlying distribution. That is, using holdout-CV to estimate model performance on such a small dataset can be susceptible to how you split your data. You can test this by re-running the pipeline with a different train/test-split and see if results ...


4

The short answer is that the y_original and x_reduced are still connected to each other, so it is safe to train your data using y_original and x_reduced. While x_reduced is on a different scale, as you mentioned via eigenvectors, it still is representative of the data that was attached to that observation, just in a different format. You lose a lot of ...


4

Yes. Clustering algorithms such as K-means do need feature scaling before they are fed to the algo. Since, clustering techniques use Euclidean Distance to form the cohorts, it will be wise e.g to scale the variables having heights in meters and weights in KGs before calculating the distance.


4

The fact that the coefficients of hp and disp are low when data is unscaled and high when data are scaled means that these variables help explaining the dependent variable but their magnitude is large, so the coefficients in the unscaled case have to be low. In terms of "importance", I would say that the absolute value of the coefficients in the scaled case ...


Only top voted, non community-wiki answers of a minimum length are eligible