Questions tagged [normalization]

The tag has no usage guidance, but it has a tag wiki.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Normalization in production

I am currently writing a machine learning pipeline for my time series application. At the end of each month, I get the data gathered, normalize it ([0, 1]), retrain the ML model with the new ...
0
votes
0answers
18 views

How to do normalize distribution technique on a simple matrix?

This is just a practice about normal distribution in MATLAB and not a real project. I created a simple matrix and want to test normal distribution on it, it's ...
0
votes
1answer
18 views

How to set and check normal distribution on a data set?

Sorry if my question is simple, I have a data set with two class and want to check and set normal distribution on it(if it was necessary, in MATLAB). But the question is that I should use it on every ...
1
vote
0answers
12 views

should I normalise data of disparate frequencies

The dataset consists of rows of time series data. Most of the time data remains constant and only updates when there are significant changes. Each time series looks something like this where each ...
0
votes
1answer
35 views

How to scale a variable when not knowing the maximum

I have a dataset with different features where some of them are not categorical, so they need to be scaled or normalized (especially the target). However, normalizing between 0-1 for instance means ...
0
votes
2answers
32 views

Multi-class classification with mostly zero valued data

I implemented a multi-class regression and wanted to test it using the MNIST dataset. I realized that if I use standardization $X \leftarrow \frac{X-mean(X)}{std(X)}$, over 50% of all features ...
0
votes
0answers
7 views

What is “style normalization” referred to in the Adaptive Instance Normalization paper?

In "Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization", the authors argue that the significant performance boost from instance normalization is not only due to contrast ...
0
votes
0answers
6 views

BatchNorm vs InstNorm from the perspective of feature distributions

What I understand so far... The main purpose of BatchNorm is to overcome covariance shift -- more specifically what the authors of BatchNorm coined "internal covariance shift". Covariance shift is ...
0
votes
0answers
16 views

PCA and orange software

I am analysing if 15 books can be grouped according to 6 variables (of the 15 books, 2 are written by an author, 6 by an other one, and 7 by an other one). I counted the number of occurrences of the ...
1
vote
1answer
21 views

Normalisation results in R^2 score of 0 - Lasso regression

I am running a regression analysis on a 7000 row dataset with a train/test split of 70%/30%. I am using one variable X to predict a variable ...
0
votes
0answers
16 views

normalization/standardization of input/output of autoencoder and Gaussian Process

I have two machine learning algorithms that deal with time series data. My data consist of 1500 time series, each of 500 time components. The first machine learning algorithm is an autoencoder, ...
0
votes
0answers
5 views

SVM, which range to use when normalizing

I am using the SVM classifier from Scikit Learn. I was wondering is there is a know-best-practice when it comes to normalization. I'm using different normalization tecniques, but all my normalized ...
0
votes
1answer
87 views

Correcting for one of multiple strong batch effects in a dataset

I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it ...
1
vote
1answer
23 views

When would not normalizing input values have higher accuracy?

Right now I'm training a deep neural network for a binary classification problem, with a feature set of winrates. As such, each winrate is bigger or equal to 0 but smaller than 100. I've been getting ...
1
vote
1answer
138 views

can I use z-score normalization even if it doesn't make sense for my data to be negative?

I'm planning to use z-score as a Normalization Method for a Project but I noticed if I do that then I ll have a data in the range [-1, 1] which is wierd because I have data that doesn't make sense for ...
0
votes
1answer
37 views

What is the use of fit method in sklearn.preprocessing.Normalizer()?

According to the documentation of fit(self, X[, y]) method of sklearn.preprocessing.Normalizer(), it does nothing and return the estimator unchanged. I understand that if I intend to normalize data I ...
0
votes
0answers
7 views

Log or StandardScaler for 0.0:1.0 values

If I wanted to normalize a bunch of values that were between zero and 1, where some values were dangerously close to zero such as 0.00001 and others were magnitudes larger like 0.8 - would I take the ...
0
votes
0answers
13 views

Is this interpretation of spectral normalisation mathematically correct?

Hello everyone, this is my first post. I was thinking about the mathematical interpretation for spectral normalization in neural networks the other day, and I came up with an explanation that feels ...
0
votes
1answer
47 views

Is it necessary to normalise data for XGBoost?

MinMaxScaler in scikit_learn is used for data normalization (a.k.a feature scaling). Data normalisation is not necessary for decision trees. Since XGBoost is based on decision trees, is it necessary ...
0
votes
1answer
82 views

If I have negative and positive numbers for a feature, should MinMaxScaler be -1 to 1?

I have a variable X with values ranging from -150 to 400. All the other variables in my training set are positive so I normalized them to be from 0 to 1, or they’re already binary, or they had a ...
0
votes
0answers
15 views

Normalization of Distribution of numeric features and target

Here, I am talking about the Normalization of Distribution of Features/Target, not normalization/scaling of them. I've been seeing Notebooks which normalize some features and the target variable, ...
0
votes
2answers
54 views

How to normalize a data set of multiple time series?

I have the a data set representing the electricity consumption of 25 000 customer. The electricity readings are taken from each smart meter each 15 min for a period of 3 days. The data is takes from ...
1
vote
1answer
36 views

Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance?

I am using raw data set with 4 feature variables to do a Binominal Classification using Logistic Regression Algorithm. I made sure that the class counts are balanced. i.e., an equal number of ...
3
votes
3answers
93 views

Why models performs better If normalize test data and train data separately?

Many threads (and courses) such as this and this one suggest that you should apply normalization to the test data using the parameters used in the training set. But other some discussions I've found ...
2
votes
0answers
42 views

Comparing feature importance in LightGBM + Scikit

I have a model trained using LightGBM (LGBMRegressor), in Python, with scikit-learn. On a weekly basis the model in re-trained, and an updated set of chosen features and associated ...
0
votes
4answers
70 views

What is the purpose of standardization in machine learning?

I'm just getting started with learning about K-nearest neighbor and am having a hard time understanding why standardization is required. Reading through, I came across a section saying When ...
2
votes
1answer
21 views

How to normalize a position in 3d game?

I'm pretty new to anything and everything related to this kind of stuff, I was wondering how would I normalize the coordinates of a entity in my game for a nerual network? Would it just be the same ...
0
votes
0answers
20 views

Is it necessary to perform Z-score or Min-Max normalization on L1-normalized data?

I have a dataset which contains vectors that generated from subtitles and have been L1 normalised, I want to calculate cosine similarity & Euclidean distance, I thought it is better if I use Z-...
0
votes
0answers
12 views

QuantileTransform before or after long-to-wide format transformation?

I perform the following steps: Perform quantile transform Transform it from wide to long format Train the anomaly detection model on the wide-format data Will there be some significant change if I, ...
0
votes
0answers
27 views

How can I reduce the noise of prediction graph?

I am trying to use LSTM to predict a time series data as you can see in the following image, the predicted graphs is very noisy: The original data is looking like this: That I normalized it like ...
0
votes
2answers
121 views

Should we denormalize our data after normalization?

If we use sklearn library's preprocessing.normalie() function to normalize our data before learning, like this: ...
0
votes
0answers
11 views

Normalising energy data using weather data?

I'm currently doing some research into energy use, and normalising it is critical for comparison across years. I'm considering ways to normalise the daily data (kWh) so that I can run statistical ...
0
votes
0answers
29 views

What is the difference between row-wise and column-wise Z-score normalization?

I have a data set, each row represents a movie name, each column is a feature (such as genres), I want to perform cosine similarity to find out the similarity between each movie, before that I need to ...
1
vote
0answers
20 views

How can detect and highlight outliers by using gaussian function and normalize the data elegantly?

I tried to normalize the data by using Gaussian function 2 times on both positive and negative numbers of each parameter of this dataset. The dataset includes missing data as well. The problem is I ...
0
votes
0answers
12 views

How to normalize old accumulated ratings

I am playing around with data-set which contains movies and their ratings by various users. I am trying to rank these movies based on their user ratings. However, as obvious, many of the old movies (...
2
votes
1answer
120 views

How to standardize my data (Univariate Time Series Forecasting using Keras LSTM)?

Let be $X = (X_1,...., X_n)$ an univariate time serie. I would like to know how to standardize my data when I split it into train and test data. Let me explain you how I tranform $X$ so that I can fit ...
1
vote
0answers
36 views

Does feature normalization improve performance of Hidden Markov Models?

For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be: Normalize to 0-mean and unit-variance ...
6
votes
3answers
1k views

Data normalization before or after train-test split?

Which one is the right approach to make data normalization - before or after train-test split? Normalization before split ...
1
vote
1answer
28 views

How many normalization methods are there and what are they for?

this post lists 5 types of normalization. Zscore MinMax Logistic LogNormal TanH is there any other types of normalization that are most commonly used in machine ...
0
votes
2answers
96 views

Would K-means be Appropriate to Use with Four or More Variables?

Just a general question that I'm trying to mentally visualize. I'm fairly new to using k-means clustering and have used it before on two variables, which creates a 2-D plot of points. I also know, ...
1
vote
1answer
113 views

is there a way to normalize [-3,1] to ${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\sqrt{10}}\\ \end{bmatrix}}$ with python?

I am learning SVD by following this MIT course The lecturer is trying to normalize a vector $${\begin{bmatrix} -3\\ 1\\ \end{bmatrix}}$$ to $${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\...
4
votes
4answers
428 views

Should input images be normalized to -1 to 1 or 0 to 1

Many ML tutorials are normalizing input images to value of -1 to 1 before feeding them to ML model. The ML model is most likely a few conv 2d layers followed by a fully connected layers. Assuming ...
0
votes
1answer
51 views

How to normalize test data according to the training data if the normalization on the training data is performed row wise?

I read on several places about the normalization of features in the machine learning method. But I normalize my training data row-wise as shown in the following ...
0
votes
2answers
149 views

Should we normalize test data by choosing maximum and minimum value of training data?

I'm training my CNN network with one model's data whereas i'm testing it with another model's data. I perform min max normalization on each sample. And every samples ranges [-1,1]. My question is that ...
0
votes
1answer
50 views

How to scale exponential data for a regression problem?

I understand that I should be scaling features between (0, 1) before feeding them into a neural network. However, what happens if future data could be larger than my current training data? For ...
3
votes
0answers
28 views

A Deep CNN model delivering better results with standardization, when compared with normalization

I developed a deep CNN model, based on the architecture discussed in this paper, to generate predictions for time series data. My training data is shown in the figure below: In order to train the ...
6
votes
2answers
537 views

Which comes first? Multiple Imputation, Splitting into train/test, or Standardization/Normalization

I am working on a multi-class classification problem, with ~65 features and ~150K instances. 30% of features are categorical and the rest are numerical (continuous). I understand that standardization ...
2
votes
1answer
184 views

MinMaxScaler when LSTM predictions fall outside of training range?

I am using MinMaxScaler on my training set and applying the transformations to my test set and inverse_transform to my model’s outputs. If this were, say, a stock prediction problem, my training set ...
1
vote
1answer
47 views

Softmax function result for already normalized probabilities

Isn't the aim of softmax function normalizing the probabilities such that they all sum to 1? So when we apply this method to the already normalized numbers, it would change them. what do these new ...
1
vote
1answer
48 views

Scaling features separately?

I have some features which are in the thousands, which I scale to the max values of these. This solves the general scaling problems, as well as preserves an important absolute value relationship ...