41

The normal vs uniform init seem to be rather unclear in fact. If we refer solely on the Glorot's and He's initializations papers, they both use a similar theoritical analysis: they find a good variance for the distribution from which the initial parameters are drawn. This variance is adapted to the activation function used and is derived without explicitly ...


29

Layer normalization (Ba 2016): Does not use batch statistics. Normalize using the statistics collected from all units within a layer of the current sample. Does not work well with ConvNets. Recurrent Batch Normalization (BN) (Cooijmans, 2016; also proposed concurrently by Qianli Liao & Tomaso Poggio, but tested on Recurrent ConvNets, instead of RNN/LSTM):...


25

This is called unity-based normalization. If you have a vector $X$, you can obtain a normalized version of it, say $Z$, by doing: $$Z = \frac{X - \min(X)}{\max(X) - \min(X)}$$


25

Lat long coordinates have a problem that they are 2 features that represent a three dimensional space. This means that the long coordinate goes all around, which means the two most extreme values are actually very close together. I've dealt with this problem a few times and what I do in this case is map them to x, y and z coordinates. This means close points ...


21

I disagree with the other comments. First of all, I see no need to normalize data for decision trees. Decision trees work by calculating a score (usually entropy) for each different division of the data $(X\leq x_i,X>x_i)$. Applying a transformation to the data that does not change the order of the data makes no difference. Random forests are just a ...


19

Normalization across instances should be done after splitting the data between training and test set, using only the data from the training set. This is because the test set plays the role of fresh unseen data, so it's not supposed to be accessible at the training stage. Using any information coming from the test set before or during training is a ...


12

That paper gives a nice answer, where i quoted from. Search for Should I standardize the target variables (column vectors)? in that page. Standardizing target variables is typically more a convenience for getting good initial weights than a necessity. However, if you have two or more target variables and your error function is scale-sensitive like the ...


11

Always split before you do any data pre-processing. Performing pre-processing before splitting will mean that information from your test set will be present during training, causing a data leak. Think of it like this, the test set is supposed to be a way of estimating performance on totally unseen data. If it affects the training, then it will be partially ...


9

The questions of whether and why it's important, depends on the context. For gradient boosted decision trees, for example, it is not important - these ML algorithms "don't care" about monotone transformations to the data; they just look for points to split it. For linear predictors, for example, scaling can improve interpretability of the results. If you'd ...


9

They are used for two different purposes. StandardScaler changes each feature column $f_{:,i}$ to $$f'_{:,i} = \frac{f_{:,i} - mean(f_{:,i})}{std(f_{:,i})}.$$ Normalizer changes each sample $x_n=(f_{n,1},...,f_{n,d})$ to $$x'_n = \frac{x_n}{size(x_n)},$$ where $size(x_n)$ for l1 norm is $\left \| x_n \right \|_1=|f_{n,1}|+...+|f_{n,d}|$, l2 norm is $\...


7

Boosting trees is about building multiple decision trees. Decision tree doesn't require feature normalization, that's because the model only needs the absolute values for branching. Wikipedia for decision tree: Requires little data preparation. Other techniques often require data normalization.... However, it's always a good idea to normalize your ...


7

You have already computed that, but you've not bound the output to a variable, also called name in python. Try the following snippet: result = np.linalg.norm(v1,ord=2,axis=1,keepdims=True) print(result) Based on the edit, I update the answer. As you may find answers to your question, a typical way to find what you need is something like the following ...


6

Yes, you should do this. Given the initialization schemes and normalized inputs, the expected values for the outputs are 0. This means that you will not be too far off from the start, which helps convergence. If your target is 1000, your mean squared error will be huge which means your gradients will also be huge which can lead to numerical instabiliy.


6

Also please explain what this array - array.mean() do? Basically, it is doing memberwise subtraction operation after broadcasting. np.mean function finds the mean in your array and its result will be a scalar, a single number. Your array is a numpy array and the result of the latter term is a single value as mentioned. Consequently, the single value gets ...


6

A detailed answer to the question can be found here. [...]are there times when it is not appropriate or not beneficial? Short answer: Yes and No. Yes in the terms, that it can significantly change your output of e.g. clustering algorithms. No, on the other hand, if these changes are what you want to achieve. Or to put it in the words of the author of ...


6

As the other answers previously said, in practice it doesn't have much difference which of the two you choose. However, theoretically it's better to scale your input to $[-1, 1]$ than $[0, 1]$ and I'd argue that it's even better to standardize your input (i.e. $μ=0$, $σ=1$). Let me explain why: Deep neural networks, especially in their early days, had ...


5

You should look into other estimators of location. What you want is a robust estimator, with a high break-down point. The extreme approach would be the median. But you may get more numerically interesting results with a trimmed mean. You define a threshold, say 2%. Then you remove the top 2% of votes, and the bottom 2% of votes, and take the mean only of ...


5

You can not use PCA, or at least it is not recommended, for mixed data. It is best to use Factor analysis of mixed data. You are lucky that Prince is a Python package that covers all data scenarios, borrowing from its explanation: All your variables are numeric: use principal component analysis (prince.PCA) You have a contingency table: use ...


5

You begin by asking about image normalisation, but then refer to other techniques, which I believe all fall under "image augmentation". So I will answer the more general question: how can I perform image augmentation to improve my model? I would generally say that the more augmentation you can apply, the better. A caveat to that statement is that the ...


5

Answer to your question: Do Normalization after splitting into train and test/validation. The reason is to avoid any data leakage. Data Leakage: Data leakage is when information from outside the training dataset is used to create the model. This additional information can allow the model to learn or know something that it otherwise would not know and in ...


5

When building any Machine Learning model, the only observable data you have is training data. Test data is supposed to be unobserved data, meaning that even though you might have it now, you need to act as if you didn't. When you apply normalisation, you first observe the data to get the parameters you need. As you are only supposed to be able to observe the ...


4

Predicting and scoring are two different tasks. And according to your answers and comments you are not solving prediction problem. You just want to set to each student a number in range [1,100] according to some rule. This is ranking (or scoring, whatever). Therefore, the terms #prediction_model, #accuracy, #validation, #training_set are out of this scope. ...


4

Yes. Feature scaling can completely change the clustering result. People usually scale data to [0:1] or to have a standard deviation of 1. However, that is nothing but a heuristic. In many cases, the need for scaling is nothing but a symptom, caused by inappropriate data for the method. You can't just fix this by some naive scaling, but it's just a hack ...


4

Standardizing (subtracting mean and dividing by standard deviation for each column), can be done using numpy: Xz = (X - np.nanmean(X, axis=0))/np.nanstd(X, axis=0) where X is a matrix (containing NaNs), and Xz is the standardized version of X. Hope this helps. EDITED: For a test/training scenario, the mean and std could be stored in respective variables:...


4

This is no longer the case; as of sklearn 0.20.0, missing values are ignored in such preprocessors' fit and silently passed along in their transform: https://scikit-learn.org/stable/whats_new.html#id82 (fourth bullet) https://github.com/scikit-learn/scikit-learn/issues/10404


4

If you would use the scaler on the full dataset you would provide the algorithm with some information about the values in the test set that it would not have otherwise. This additional information ("data leakage") from your test set to your training set is problematic because it leads to unreliable performance estimates. It is therefore not surprising that ...


4

I don't understand why you would like to fill values with zeros ! This would basically mean, "this guy, who is 170 cm tall, weights 0 kg" and would fool your network. In my opinion, you have two options: discard missing values (the entire row): you end up with less but more consistent training data if you really need these rows, then fill missing values ...


4

As @Erwan said, you should normalize the training set and then use the same normalization steps on the test set. So your code should look like: x_train, x_test, y_train, y_test = train_test_split(X_features, Y_feature, test_size=0.20, random_state=4) scaler = StandardScaler() normalized_x_train = pd.DataFrame(scaler.fit_transform(x_train), columns = ...


4

If apply normalization on training and testing in a separate way, I get really good results 85% (and sometimes more) and the further steps I try to do next work better as well. The problem with applying normalization across instances on the test set separately is that the test set represents any new data. So in principle the model should be able to give a ...


4

The minimum and maximum values are just known limits that are parts of the formula that reshapes the distribution of the data, so if a value is bigger than the previously known value the resulting feature scaling (Normalization) will be still appropriate. An alternative is z-scores if you don't feel like using minimum and maximum values. x'= (x-x̄) / σ ...


Only top voted, non community-wiki answers of a minimum length are eligible