Questions tagged [normalization]

The tag has no usage guidance, but it has a tag wiki.

Filter by
Sorted by
Tagged with
0
votes
1answer
9 views

Should I normalise my data if future unseen data may have a different range?

I'm new to ML and researching data prep, more specifically feature normalisation. My question is whether it's a good idea to normalise data when its range may change over time? For example, if I'm ...
0
votes
1answer
12 views

Normalize data between 0 and 95 instead of between 0 and 100

I want to normalize the data between 0 and 95 instead of 0 and 100. I am using this formula to normalize between 0 and 100, please let me know how to edit it. ...
1
vote
1answer
113 views

Data normalization of count data for neural networks

I have a sparse matrix of count data that I'm using as input to a neural network. I know, usually, the input data should be normalized (e.g. via min-max scaling, $z$-score standardization, etc.). But ...
0
votes
1answer
13 views

How should a stateless data transformation be applied in regard to train/test split?

I want to apply spatial sign transformation to my data, but unlike other transformations this one is stateless. I am using sklearn and normallly i would first use ...
0
votes
1answer
26 views

K-means clustering with categorical data

I am doing a clustering analysis using K-means and I have around 6 categorical variables that I want to consider in the model. When I transform these variables as dummy variables (binary values 1 - 0) ...
2
votes
1answer
223 views

Normalization in production

I am currently writing a machine learning pipeline for my time series application. At the end of each month, I get the data gathered, normalize it ([0, 1]), retrain the ML model with the new ...
0
votes
0answers
20 views

Is it better to normalize raw data to [-1, 1]

I am working on Radio signal classification. I have data generated synthetically and not normalized with the following features: ...
2
votes
2answers
140 views

Multi-class classification with mostly zero valued data

I implemented a multi-class classification and wanted to test it using the MNIST dataset. I realized that if I use standardization $X \leftarrow \frac{X-mean(X)}{std(X)}$, over 50% of all features ...
1
vote
1answer
81 views

Should output data scaling correspond to the activation function's output?

I am building an LSTM with keras which have an activation parameter in the layer. I have read that scaling on the output data ...
0
votes
1answer
42 views

What is the best practice to normalize/standardize imbalanced data for outlier detection or binary classification task?

I'm researching Anomaly/outlier/fraud detection, and I'm looking for the best practice to pre-process the synthetic data for imbalanced data. I have checked all methodology for normalizing/...
1
vote
0answers
23 views

Implementation of an Adaptive Normalization method

Referencing: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.718.9985&rep=rep1&type=pdf I'm trying to wrap my head around the method described in this paper I currently have 2 main ...
2
votes
1answer
40 views

Normalization vs standardization for image classification problem

For day and night image classification, is it better to normalize or standardize images? In general, when should I use each method? I am interested in with example why one method is preferred over ...
1
vote
1answer
98 views

First perform data augmentation or normalization?

Should I first perform data augmentation or normalization in deep learning? I am mainly interested in 2D and 3D input data. In tutorials that I have seen so far the data augmentation always comes ...
3
votes
1answer
210 views

Data normalization in nonstationary data classification with Learn++.NSE based on MLP

I need to predict technical aggregate condition using vibration monitoring data. We consider this data to be nonstationary i.e. distribution parameters and descriptive statistics are not constant. I ...
0
votes
1answer
114 views

PCA and orange software

I am analysing if 15 books can be grouped according to 6 variables (of the 15 books, 2 are written by an author, 6 by an other one, and 7 by an other one). I counted the number of occurrences of the ...
1
vote
1answer
40 views

Why my regression model always be dominanted by one feature?

I am working on a financial predict problem. which means it is a time series prediction problem. I have three features, which have high correlation(each two's corr is about 0.6) And I do the linear ...
0
votes
0answers
5 views

How to Normalize image intensities of CT and MRI images (single channel)

I have data-set which contains MRI and CT images and all of them are labeled. I want to create MRI-CT classifier. But intensity range of MRI and CT are different. CT ranges between (-1024 and 2000) ...
0
votes
0answers
24 views

Input pipeline with an autoencoder and tf.data

I am using an autoencoder to detect anomalies in dataset of network traffic. The dataset is a csv file, and is loaded and preprocessed with pandas (encoded categorical features with pandas.get_dummies(...
0
votes
1answer
11 views

Normalization of possibly not fully representative data

I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:...
3
votes
1answer
50 views

What parameters to use when normalising training, validation, and testing data?

I know a similar post was made here, but I wanted to ask some follow up questions. I am conducting a cross-validation search to find values of a set of hyper-parameters and need to normalise the data. ...
0
votes
1answer
39 views

Is it possible to do Normalization before Xgboost?

Currently I am working on a project which uses Xgboost Regression. Before putting data into model, I implemented Normalization, the accuracy significantly increased compared with without Normalization....
1
vote
1answer
6k views

Standardization/Normalization test data in R

I understand that one should standardize and normalize the test data (or any "unlabeled" data) with the training mean and sd. How can I implement this in R language? Is there a kind of "fitting" to ...
1
vote
0answers
26 views

k-means and unequal ranges

I am trying to cluster an image using k-means clustering algorithm. However, the features that I use (color, position) have different ranges, for that I notice the clustering is biased for the feature ...
1
vote
1answer
104 views

Correcting for one of multiple strong batch effects in a dataset

I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it ...
1
vote
1answer
226 views

Should Principal components be normalized before applying K means on them?

I want to get the Principal components of a dataset and apply K mean clustering on them. Do I need to Normalized the PCA output before applying Kmeans on them ?
2
votes
1answer
31 views

Is feature scaling at all needed for a feature set with a single feature?

I understand that feature scaling is required to bring features in different magnitudes on a common scale so the model is not biased towards features with higher magnitudes. But if there is only a ...
0
votes
0answers
23 views

Normalization during Inference in Keras

I am training a binary neural network so I can obtain its weights, and use them on another network where I add a few things. For example, this this one layer of the network: ...
1
vote
1answer
62 views

What is the difference between normalization and re-scaling?

This site does not describe the nature of the normalization tag. How does it differ from re-scaling? Many authors use the two terms interchangeably. I can not understand normalization's operational ...
1
vote
1answer
86 views

Generating the right target for an LSTM model

Trying to explain my question on a simplified data set. Having the following dataset: ...
1
vote
2answers
708 views

Time series normalization using min max technique

I have a time series dataset and I would like to normalize the data (diff which is of type list) as below using Min Max ...
0
votes
0answers
30 views

Normalization in Neural Networks Regression in MATLAB

I have 20 samples for a regression fitting purpose in Neural Network Toolbox (nnstart, nftool) in MATLAB . I have 3 inputs of 20 samples each (3 X 20 ) and 1 output (3 X 20, as the output number is ...
0
votes
1answer
306 views

How to normalize test data according to the training data if the normalization on the training data is performed row wise?

I read on several places about the normalization of features in the machine learning method. But I normalize my training data row-wise as shown in the following ...
0
votes
1answer
158 views

Use both differencing and normalization in time series modeling to make it stationary?

I am working on a time series dataset. Should we use both differencing and normalizing or either of the ones to make it stationary?
0
votes
2answers
1k views

Should we normalize test data by choosing maximum and minimum value of training data?

I'm training my CNN network with one model's data whereas i'm testing it with another model's data. I perform min max normalization on each sample. And every samples ranges [-1,1]. My question is that ...
15
votes
4answers
40k views

How to scale an array of signed integers to range from 0 to 1?

I'm using Brain to train a neural network on a feature set that includes both positive and negative values. But Brain requires input values between 0 and 1. What's the best way to normalize my data?
15
votes
2answers
10k views

Is it necessary to normalise data for XGBoost?

MinMaxScaler in scikit_learn is used for data normalization (a.k.a feature scaling). Data normalisation is not necessary for decision trees. Since XGBoost is based on decision trees, is it necessary ...
1
vote
1answer
42 views

Normalize data from different groups

I have data that have been grouped in 27 groups by different criteria. The reason for these groupings is to show that each group has a different behavior. However, I would like to normalize everything ...
1
vote
0answers
80 views

Sliding Window Normalization

I've been reading about time series forecasting and many approaches use a sliding window method. What I don't get about it is how to properly normalize your data. Most codes I've seen so far normalize ...
1
vote
0answers
26 views

Normalizing variables with logarithmic shape

A simple model with two variables [A,B] to train, let's say, a logistic regression or any other classification model: A: Flat distribution from 0 to 100. B: A logarithmic distribution from 0 to a few ...
0
votes
1answer
17 views

Normalized 2D tensor values are not in range 0-1

Below function takes in 2D tensor and normalizes it using broadcasting .The issue is except all values to be in range 0-1 but the result has values outside this range . How to get all values in 2D ...
1
vote
1answer
42 views

Why does my manual derivative of Layer Normalization imply no gradient flow?

I recently tried computing the derivative of the layer norm function (https://arxiv.org/abs/1607.06450), an essential component of transformers, but the result suggests that no gradient flows through ...
1
vote
0answers
33 views

Considerations to take into account when clustering

My idea is to use clustering to perform stock segmentation based on risk, building different risk levels that might adapt better to different kind of users. Hence I have computed different risk ...
0
votes
2answers
48 views

SVM, which range to use when normalizing

I am using the SVM classifier from Scikit Learn. I was wondering is there is a know-best-practice when it comes to normalization. I'm using different normalization tecniques, but all my normalized ...
1
vote
0answers
14 views

Derivative of the function of random variable

Suppose we have a function $\phi(X)$ of random variable $X$. Suppose both of $\phi(X)$ and $X$ are random variables. If $\phi$ is differentiable, how to calculate the derivative of $\phi(X)$ w.r.t. $...
0
votes
1answer
891 views

Subtracting grand mean from train and test images

I am building an image classifier based off the VGG_face keras implementation. It is easiest for me to extract a csv file full of the representations and then try classifiers on those representations. ...
6
votes
1answer
6k views

It is helpful to normalize target variables for a regression neural network?

It is customary to normalize feature variables and this normally does increase the performance of a neural network in particular a CNN. I was wondering if normalizing the target could also help ...
0
votes
0answers
35 views

Why does normalization improve my decision tree performances?

I have a regression problem for which I have to try several models, so I normalized my data and then tried to use a decision tree regressor (from sklearn.tree) and I noticed very good results (...
3
votes
2answers
145 views

Layer normalization details in GPT-2

I've read that GPT-2 and other transformers use layer normalization before the self-attention and feedforward blocks, but I am still unsure exactly how the normalization works. Let's say that our ...
0
votes
1answer
41 views

Normalization for a 2d input array

I am new to machine learning and trying to apply it to my problem. I have a training dataset with 44000 rows of features with shape 6, 25. I want to build a sequential model. I was wondering if there ...

1
2 3 4 5