A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [normalization]

The tag has no usage guidance, but it has a tag wiki.

Filter by
Sorted by
Tagged with
1
vote
1answer
21 views

Does it make sense to normalize?

My data consists of multiple NHL seasons where the total number of goals scored by each player is kept track. I intend to compare the distribution of goals scored between each season. My thinking ...
2
votes
1answer
27 views

Batch Normalization vs Other Normalization Techniques

In the context of neural networks, I understand that batch normalization ensures that activation at each layer of the neural net does not 'blow-up' and cause a bias in the network. However, I don't ...
2
votes
1answer
36 views

Differences between normalization and standarization in multiple regression

Consider the following question regarding multiple regression 1) Can someone explain why we have to transform dependent variable using log-transformation (Normalization) when appear positive skewed y ...
1
vote
3answers
39 views

why to use Scaler.fit only on x_train and not on x_test for normalizing value using MinMaxScaler?

while normalising the data everone is saying that we need to fit only on x_train and not on x_test ? why is that we should not fit x_test ? if we should not fit the scaler on x_test then why we need ...
1
vote
1answer
14 views

How to append new numerical features to an embedding from word2vec, such that KNN on embeddings is not biased for one feature?

I am working on similarity calculation between entities of similar type. For each entity I am able to make a vector that comprises of multiple vectors itself. A = 50*1 vector B = 100*1 vector C = 50*...
0
votes
1answer
32 views

CrossMapLRN2d in pytorch

I had to convert a code written in pytorch to keras (with tensorflow backend). But there was this layer called CrossMapLRN2d which had no direct counterpart in Keras. So wanted to know what does this ...
0
votes
1answer
48 views

why does making the target variable normally distributed helps?

while working on some regression problems I have found that if the target variable is skewed, making it normally distributed(using transformations) almost always helps. Why is that? Should we also ...
1
vote
1answer
27 views

Normalization in production

I am currently writing a machine learning pipeline for my time series application. At the end of each month, I get the data gathered, normalize it ([0, 1]), retrain the ML model with the new ...
0
votes
0answers
21 views

How to do normalize distribution technique on a simple matrix?

This is just a practice about normal distribution in MATLAB and not a real project. I created a simple matrix and want to test normal distribution on it, it's ...
0
votes
1answer
19 views

How to set and check normal distribution on a data set?

Sorry if my question is simple, I have a data set with two class and want to check and set normal distribution on it(if it was necessary, in MATLAB). But the question is that I should use it on every ...
1
vote
0answers
12 views

should I normalise data of disparate frequencies

The dataset consists of rows of time series data. Most of the time data remains constant and only updates when there are significant changes. Each time series looks something like this where each ...
0
votes
1answer
36 views

How to scale a variable when not knowing the maximum

I have a dataset with different features where some of them are not categorical, so they need to be scaled or normalized (especially the target). However, normalizing between 0-1 for instance means ...
0
votes
2answers
41 views

Multi-class classification with mostly zero valued data

I implemented a multi-class regression and wanted to test it using the MNIST dataset. I realized that if I use standardization $X \leftarrow \frac{X-mean(X)}{std(X)}$, over 50% of all features ...
0
votes
0answers
16 views

What is “style normalization” referred to in the Adaptive Instance Normalization paper?

In "Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization", the authors argue that the significant performance boost from instance normalization is not only due to contrast ...
0
votes
0answers
6 views

BatchNorm vs InstNorm from the perspective of feature distributions

What I understand so far... The main purpose of BatchNorm is to overcome covariance shift -- more specifically what the authors of BatchNorm coined "internal covariance shift". Covariance shift is ...
0
votes
0answers
19 views

PCA and orange software

I am analysing if 15 books can be grouped according to 6 variables (of the 15 books, 2 are written by an author, 6 by an other one, and 7 by an other one). I counted the number of occurrences of the ...
1
vote
1answer
21 views

Normalisation results in R^2 score of 0 - Lasso regression

I am running a regression analysis on a 7000 row dataset with a train/test split of 70%/30%. I am using one variable X to predict a variable ...
0
votes
0answers
19 views

normalization/standardization of input/output of autoencoder and Gaussian Process

I have two machine learning algorithms that deal with time series data. My data consist of 1500 time series, each of 500 time components. The first machine learning algorithm is an autoencoder, ...
0
votes
1answer
8 views

SVM, which range to use when normalizing

I am using the SVM classifier from Scikit Learn. I was wondering is there is a know-best-practice when it comes to normalization. I'm using different normalization tecniques, but all my normalized ...
0
votes
1answer
89 views

Correcting for one of multiple strong batch effects in a dataset

I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it ...
1
vote
1answer
23 views

When would not normalizing input values have higher accuracy?

Right now I'm training a deep neural network for a binary classification problem, with a feature set of winrates. As such, each winrate is bigger or equal to 0 but smaller than 100. I've been getting ...
1
vote
1answer
144 views

can I use z-score normalization even if it doesn't make sense for my data to be negative?

I'm planning to use z-score as a Normalization Method for a Project but I noticed if I do that then I ll have a data in the range [-1, 1] which is wierd because I have data that doesn't make sense for ...
0
votes
1answer
55 views

What is the use of fit method in sklearn.preprocessing.Normalizer()?

According to the documentation of fit(self, X[, y]) method of sklearn.preprocessing.Normalizer(), it does nothing and return the estimator unchanged. I understand that if I intend to normalize data I ...
0
votes
0answers
9 views

Log or StandardScaler for 0.0:1.0 values

If I wanted to normalize a bunch of values that were between zero and 1, where some values were dangerously close to zero such as 0.00001 and others were magnitudes larger like 0.8 - would I take the ...
0
votes
0answers
13 views

Is this interpretation of spectral normalisation mathematically correct?

Hello everyone, this is my first post. I was thinking about the mathematical interpretation for spectral normalization in neural networks the other day, and I came up with an explanation that feels ...
0
votes
1answer
118 views

Is it necessary to normalise data for XGBoost?

MinMaxScaler in scikit_learn is used for data normalization (a.k.a feature scaling). Data normalisation is not necessary for decision trees. Since XGBoost is based on decision trees, is it necessary ...
0
votes
1answer
169 views

If I have negative and positive numbers for a feature, should MinMaxScaler be -1 to 1?

I have a variable X with values ranging from -150 to 400. All the other variables in my training set are positive so I normalized them to be from 0 to 1, or they’re already binary, or they had a ...
0
votes
0answers
16 views

Normalization of Distribution of numeric features and target

Here, I am talking about the Normalization of Distribution of Features/Target, not normalization/scaling of them. I've been seeing Notebooks which normalize some features and the target variable, ...
0
votes
2answers
54 views

How to normalize a data set of multiple time series?

I have the a data set representing the electricity consumption of 25 000 customer. The electricity readings are taken from each smart meter each 15 min for a period of 3 days. The data is takes from ...
1
vote
1answer
39 views

Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance?

I am using raw data set with 4 feature variables to do a Binominal Classification using Logistic Regression Algorithm. I made sure that the class counts are balanced. i.e., an equal number of ...
3
votes
3answers
102 views

Why models performs better If normalize test data and train data separately?

Many threads (and courses) such as this and this one suggest that you should apply normalization to the test data using the parameters used in the training set. But other some discussions I've found ...
2
votes
0answers
60 views

Comparing feature importance in LightGBM + Scikit

I have a model trained using LightGBM (LGBMRegressor), in Python, with scikit-learn. On a weekly basis the model in re-trained, and an updated set of chosen features and associated ...
0
votes
4answers
92 views

What is the purpose of standardization in machine learning?

I'm just getting started with learning about K-nearest neighbor and am having a hard time understanding why standardization is required. Reading through, I came across a section saying When ...
2
votes
1answer
22 views

How to normalize a position in 3d game?

I'm pretty new to anything and everything related to this kind of stuff, I was wondering how would I normalize the coordinates of a entity in my game for a nerual network? Would it just be the same ...
0
votes
0answers
23 views

Is it necessary to perform Z-score or Min-Max normalization on L1-normalized data?

I have a dataset which contains vectors that generated from subtitles and have been L1 normalised, I want to calculate cosine similarity & Euclidean distance, I thought it is better if I use Z-...
0
votes
0answers
13 views

QuantileTransform before or after long-to-wide format transformation?

I perform the following steps: Perform quantile transform Transform it from wide to long format Train the anomaly detection model on the wide-format data Will there be some significant change if I, ...
0
votes
0answers
33 views

How can I reduce the noise of prediction graph?

I am trying to use LSTM to predict a time series data as you can see in the following image, the predicted graphs is very noisy: The original data is looking like this: That I normalized it like ...
0
votes
2answers
197 views

Should we denormalize our data after normalization?

If we use sklearn library's preprocessing.normalie() function to normalize our data before learning, like this: ...
0
votes
0answers
12 views

Normalising energy data using weather data?

I'm currently doing some research into energy use, and normalising it is critical for comparison across years. I'm considering ways to normalise the daily data (kWh) so that I can run statistical ...
0
votes
0answers
36 views

What is the difference between row-wise and column-wise Z-score normalization?

I have a data set, each row represents a movie name, each column is a feature (such as genres), I want to perform cosine similarity to find out the similarity between each movie, before that I need to ...
1
vote
0answers
24 views

How can detect and highlight outliers by using gaussian function and normalize the data elegantly?

I tried to normalize the data by using Gaussian function 2 times on both positive and negative numbers of each parameter of this dataset. The dataset includes missing data as well. The problem is I ...
0
votes
0answers
12 views

How to normalize old accumulated ratings

I am playing around with data-set which contains movies and their ratings by various users. I am trying to rank these movies based on their user ratings. However, as obvious, many of the old movies (...
2
votes
1answer
154 views

How to standardize my data (Univariate Time Series Forecasting using Keras LSTM)?

Let be $X = (X_1,...., X_n)$ an univariate time serie. I would like to know how to standardize my data when I split it into train and test data. Let me explain you how I tranform $X$ so that I can fit ...
1
vote
0answers
39 views

Does feature normalization improve performance of Hidden Markov Models?

For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be: Normalize to 0-mean and unit-variance ...
7
votes
3answers
2k views

Data normalization before or after train-test split?

Which one is the right approach to make data normalization - before or after train-test split? Normalization before split ...
1
vote
1answer
28 views

How many normalization methods are there and what are they for?

this post lists 5 types of normalization. Zscore MinMax Logistic LogNormal TanH is there any other types of normalization that are most commonly used in machine ...
0
votes
2answers
169 views

Would K-means be Appropriate to Use with Four or More Variables?

Just a general question that I'm trying to mentally visualize. I'm fairly new to using k-means clustering and have used it before on two variables, which creates a 2-D plot of points. I also know, ...
1
vote
1answer
119 views

is there a way to normalize [-3,1] to ${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\sqrt{10}}\\ \end{bmatrix}}$ with python?

I am learning SVD by following this MIT course The lecturer is trying to normalize a vector $${\begin{bmatrix} -3\\ 1\\ \end{bmatrix}}$$ to $${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\...
5
votes
4answers
655 views

Should input images be normalized to -1 to 1 or 0 to 1

Many ML tutorials are normalizing input images to value of -1 to 1 before feeding them to ML model. The ML model is most likely a few conv 2d layers followed by a fully connected layers. Assuming ...
0
votes
1answer
64 views

How to normalize test data according to the training data if the normalization on the training data is performed row wise?

I read on several places about the normalization of features in the machine learning method. But I normalize my training data row-wise as shown in the following ...