People who code: we want your input. Take the Survey

# Tag Info

53

The normal vs uniform init seem to be rather unclear in fact. If we refer solely on the Glorot's and He's initializations papers, they both use a similar theoritical analysis: they find a good variance for the distribution from which the initial parameters are drawn. This variance is adapted to the activation function used and is derived without explicitly ...

31

Normalization across instances should be done after splitting the data between training and test set, using only the data from the training set. This is because the test set plays the role of fresh unseen data, so it's not supposed to be accessible at the training stage. Using any information coming from the test set before or during training is a ...

30

Lat long coordinates have a problem that they are 2 features that represent a three dimensional space. This means that the long coordinate goes all around, which means the two most extreme values are actually very close together. I've dealt with this problem a few times and what I do in this case is map them to x, y and z coordinates. This means close points ...

30

Layer normalization (Ba 2016): Does not use batch statistics. Normalize using the statistics collected from all units within a layer of the current sample. Does not work well with ConvNets. Recurrent Batch Normalization (BN) (Cooijmans, 2016; also proposed concurrently by Qianli Liao & Tomaso Poggio, but tested on Recurrent ConvNets, instead of RNN/LSTM):...

26

This is called unity-based normalization. If you have a vector $X$, you can obtain a normalized version of it, say $Z$, by doing: $$Z = \frac{X - \min(X)}{\max(X) - \min(X)}$$

25

I disagree with the other comments. First of all, I see no need to normalize data for decision trees. Decision trees work by calculating a score (usually entropy) for each different division of the data $(X\leq x_i,X>x_i)$. Applying a transformation to the data that does not change the order of the data makes no difference. Random forests are just a ...

17

That paper gives a nice answer, where i quoted from. Search for Should I standardize the target variables (column vectors)? in that page. Standardizing target variables is typically more a convenience for getting good initial weights than a necessity. However, if you have two or more target variables and your error function is scale-sensitive like the ...

15

Your rationale is indeed correct: decision trees do not require normalization of their inputs; and since XGBoost is essentially an ensemble algorithm comprised of decision trees, it does not require normalization for the inputs either. For corroboration, see also the thread Is Normalization necessary? at the XGBoost Github repo, where the answer by the lead ...

14

Always split before you do any data pre-processing. Performing pre-processing before splitting will mean that information from your test set will be present during training, causing a data leak. Think of it like this, the test set is supposed to be a way of estimating performance on totally unseen data. If it affects the training, then it will be partially ...

10

Boosting trees is about building multiple decision trees. Decision tree doesn't require feature normalization, that's because the model only needs the absolute values for branching. Wikipedia for decision tree: Requires little data preparation. Other techniques often require data normalization.... However, it's always a good idea to normalize your ...

10

The questions of whether and why it's important, depends on the context. For gradient boosted decision trees, for example, it is not important - these ML algorithms "don't care" about monotone transformations to the data; they just look for points to split it. For linear predictors, for example, scaling can improve interpretability of the results. If you'd ...

10

They are used for two different purposes. StandardScaler changes each feature column $f_{:,i}$ to $$f'_{:,i} = \frac{f_{:,i} - mean(f_{:,i})}{std(f_{:,i})}.$$ Normalizer changes each sample $x_n=(f_{n,1},...,f_{n,d})$ to $$x'_n = \frac{x_n}{size(x_n)},$$ where $size(x_n)$ for l1 norm is $\left \| x_n \right \|_1=|f_{n,1}|+...+|f_{n,d}|$, l2 norm is $\... 9 Yes, you should do this. Given the initialization schemes and normalized inputs, the expected values for the outputs are 0. This means that you will not be too far off from the start, which helps convergence. If your target is 1000, your mean squared error will be huge which means your gradients will also be huge which can lead to numerical instabiliy. 8 As @Erwan said, you should normalize the training set and then use the same normalization steps on the test set. So your code should look like: x_train, x_test, y_train, y_test = train_test_split(X_features, Y_feature, test_size=0.20, random_state=4) scaler = StandardScaler() normalized_x_train = pd.DataFrame(scaler.fit_transform(x_train), columns = ... 7 This is no longer the case; as of sklearn 0.20.0, missing values are ignored in such preprocessors' fit and silently passed along in their transform: https://scikit-learn.org/stable/whats_new/v0.20.html#id37 (fourth bullet) https://github.com/scikit-learn/scikit-learn/issues/10404 7 Also please explain what this array - array.mean() do? Basically, it is doing memberwise subtraction operation after broadcasting. np.mean function finds the mean in your array and its result will be a scalar, a single number. Your array is a numpy array and the result of the latter term is a single value as mentioned. Consequently, the single value gets ... 7 You have already computed that, but you've not bound the output to a variable, also called name in python. Try the following snippet: result = np.linalg.norm(v1,ord=2,axis=1,keepdims=True) print(result) Based on the edit, I update the answer. As you may find answers to your question, a typical way to find what you need is something like the following ... 7 Answer to your question: Do Normalization after splitting into train and test/validation. The reason is to avoid any data leakage. Data Leakage: Data leakage is when information from outside the training dataset is used to create the model. This additional information can allow the model to learn or know something that it otherwise would not know and in ... 6 Standardizing (subtracting mean and dividing by standard deviation for each column), can be done using numpy: Xz = (X - np.nanmean(X, axis=0))/np.nanstd(X, axis=0) where X is a matrix (containing NaNs), and Xz is the standardized version of X. Hope this helps. EDITED: For a test/training scenario, the mean and std could be stored in respective variables:... 6 A detailed answer to the question can be found here. [...]are there times when it is not appropriate or not beneficial? Short answer: Yes and No. Yes in the terms, that it can significantly change your output of e.g. clustering algorithms. No, on the other hand, if these changes are what you want to achieve. Or to put it in the words of the author of ... 6 You begin by asking about image normalisation, but then refer to other techniques, which I believe all fall under "image augmentation". So I will answer the more general question: how can I perform image augmentation to improve my model? I would generally say that the more augmentation you can apply, the better. A caveat to that statement is that the ... 6 As the other answers previously said, in practice it doesn't have much difference which of the two you choose. However, theoretically it's better to scale your input to$[-1, 1]$than$[0, 1]$and I'd argue that it's even better to standardize your input (i.e.$μ=0$,$σ=1\$). Let me explain why: Deep neural networks, especially in their early days, had ...

5

You should look into other estimators of location. What you want is a robust estimator, with a high break-down point. The extreme approach would be the median. But you may get more numerically interesting results with a trimmed mean. You define a threshold, say 2%. Then you remove the top 2% of votes, and the bottom 2% of votes, and take the mean only of ...

5

You can not use PCA, or at least it is not recommended, for mixed data. It is best to use Factor analysis of mixed data. You are lucky that Prince is a Python package that covers all data scenarios, borrowing from its explanation: All your variables are numeric: use principal component analysis (prince.PCA) You have a contingency table: use ...

5

One reason for normalising the inputs is to make gradient descent more stable, as gradients spend more time in a comfortable region with meaningful updates and less neurons 'die' during trainings - getting stuck at one of the tails of e.g. the sigmoid non-linearity. Normalising the output distribution is perhaps not the best idea, as you are by definition ...

5

I don't understand why you would like to fill values with zeros ! This would basically mean, "this guy, who is 170 cm tall, weights 0 kg" and would fool your network. In my opinion, you have two options: discard missing values (the entire row): you end up with less but more consistent training data if you really need these rows, then fill missing values ...

5

If apply normalization on training and testing in a separate way, I get really good results 85% (and sometimes more) and the further steps I try to do next work better as well. The problem with applying normalization across instances on the test set separately is that the test set represents any new data. So in principle the model should be able to give a ...

5

When building any Machine Learning model, the only observable data you have is training data. Test data is supposed to be unobserved data, meaning that even though you might have it now, you need to act as if you didn't. When you apply normalisation, you first observe the data to get the parameters you need. As you are only supposed to be able to observe the ...

4

You can use sklearn.preprocessing.QuantileTransformer (or sklearn.preprocessing.PowerTransformer) which does exactly what you want: from sklearn.preprocessing import QuantileTransformer import numpy as np ey = np.random.exponential(size=100) qt = QuantileTransformer(output_distribution='normal') no = qt.fit_transform(ey.reshape(-1, 1)) You can plot ...

4

How is your experience using feature normalization with boosted trees does it in general improve our models? My rather limited experience with scaling of features suggests that it has virtually no impact on xgboost results. I suppose by normalisation you mean subtracting the mean and then dividing by standard deviation. If you calculated the statistics ...

Only top voted, non community-wiki answers of a minimum length are eligible