Questions tagged [normalization]

The tag has no usage guidance, but it has a tag wiki.

79 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
1answer
50 views

What parameters to use when normalising training, validation, and testing data?

I know a similar post was made here, but I wanted to ask some follow up questions. I am conducting a cross-validation search to find values of a set of hyper-parameters and need to normalise the data. ...
3
votes
0answers
36 views

A Deep CNN model delivering better results with standardization, when compared with normalization

I developed a deep CNN model, based on the architecture discussed in this paper, to generate predictions for time series data. My training data is shown in the figure below: In order to train the ...
3
votes
0answers
52 views

Standardizing data from different distributions

I have energy data from different homes (~200), and I wish to merge all the energy data to create a combined dataset. Since the individual homes have different power consumption, I get different ...
3
votes
1answer
210 views

Data normalization in nonstationary data classification with Learn++.NSE based on MLP

I need to predict technical aggregate condition using vibration monitoring data. We consider this data to be nonstationary i.e. distribution parameters and descriptive statistics are not constant. I ...
2
votes
0answers
271 views

K-Means Clustering Profile Plot & Data Normalization

I am new to k-means clustering and I am working on a project on cryptoanalysis. I have a few questions and I hope to get some help here. I have four variables and my variables data values can range ...
2
votes
1answer
223 views

Normalization in production

I am currently writing a machine learning pipeline for my time series application. At the end of each month, I get the data gathered, normalize it ([0, 1]), retrain the ML model with the new ...
2
votes
2answers
140 views

Multi-class classification with mostly zero valued data

I implemented a multi-class classification and wanted to test it using the MNIST dataset. I realized that if I use standardization $X \leftarrow \frac{X-mean(X)}{std(X)}$, over 50% of all features ...
2
votes
0answers
160 views

Comparing feature importance in LightGBM + Scikit

I have a model trained using LightGBM (LGBMRegressor), in Python, with scikit-learn. On a weekly basis the model in re-trained, and an updated set of chosen features and associated ...
2
votes
0answers
188 views

Does feature normalization improve performance of Hidden Markov Models?

For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be: Normalize to 0-mean and unit-variance ...
2
votes
0answers
37 views

Normalising time(minutes) related data with n other input variables(also dependant on time)

I am working on the GPS dataset of the football players with the energy expenditure being my output variable and other 15 variables being input/independent variables. My objective is to forecast the ...
1
vote
0answers
23 views

Implementation of an Adaptive Normalization method

Referencing: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.718.9985&rep=rep1&type=pdf I'm trying to wrap my head around the method described in this paper I currently have 2 main ...
1
vote
0answers
26 views

k-means and unequal ranges

I am trying to cluster an image using k-means clustering algorithm. However, the features that I use (color, position) have different ranges, for that I notice the clustering is biased for the feature ...
1
vote
1answer
40 views

Why my regression model always be dominanted by one feature?

I am working on a financial predict problem. which means it is a time series prediction problem. I have three features, which have high correlation(each two's corr is about 0.6) And I do the linear ...
1
vote
0answers
80 views

Sliding Window Normalization

I've been reading about time series forecasting and many approaches use a sliding window method. What I don't get about it is how to properly normalize your data. Most codes I've seen so far normalize ...
1
vote
0answers
33 views

Considerations to take into account when clustering

My idea is to use clustering to perform stock segmentation based on risk, building different risk levels that might adapt better to different kind of users. Hence I have computed different risk ...
1
vote
0answers
14 views

Derivative of the function of random variable

Suppose we have a function $\phi(X)$ of random variable $X$. Suppose both of $\phi(X)$ and $X$ are random variables. If $\phi$ is differentiable, how to calculate the derivative of $\phi(X)$ w.r.t. $...
1
vote
0answers
26 views

Normalizing variables with logarithmic shape

A simple model with two variables [A,B] to train, let's say, a logistic regression or any other classification model: A: Flat distribution from 0 to 100. B: A logarithmic distribution from 0 to a few ...
1
vote
0answers
46 views

whether to use normalization while performing PCA?

I have an excel file containing a table where I registered the frequency of three linguistic phenomena in 72 poems. Since the poems have different lengths I normalized the results dividing each value ...
1
vote
1answer
98 views

First perform data augmentation or normalization?

Should I first perform data augmentation or normalization in deep learning? I am mainly interested in 2D and 3D input data. In tutorials that I have seen so far the data augmentation always comes ...
1
vote
0answers
16 views

Rank advertisement and Score keywords used in advertisement based on its performance

I am trying to classify my advertisements and trying to understand which advertisement is performing better than others and why. Step - 1: compare two ads based on CTR. The problem here is I am ...
1
vote
0answers
31 views

Generalize min-max scaling to vectors

I am combining several vectors, where each vector is a certain kind of embedding of some object. Since each embedding is very different (some have all components between $[0, 1]$ some have components ...
1
vote
0answers
29 views

Normalisation of features extracted from audio files

I am building CNN and SVM models which take in MFCC features as input. The MFCC matrices shape is (13, n). The 13 rows are coefficients and n columns represent n time frames. So each row in the matrix ...
1
vote
1answer
86 views

Generating the right target for an LSTM model

Trying to explain my question on a simplified data set. Having the following dataset: ...
1
vote
0answers
24 views

Normalization around the nominal value while avoiding to get too big results

Lets say we measured a signal N times. Each measurement contains T time instance. For example for N=4 measurements and T=5 time instances the data matrix is as follows: ...
1
vote
0answers
18 views

When should I reverse normalizations to evaluate loss?

If I am training a neural network and have normalized the data before-hand, should I reverse the normalization to calculate the loss? This tutorial provides an example of this method. What if I'm ...
1
vote
0answers
76 views

How to handle negative delays when predicting flight delays?

I am working with the nycflights dataset. My goal is to predict departure and arrival time delays using random forests. My ...
1
vote
0answers
23 views

Normalizing historical data in time-series LSTMs

I am currently trying to solve a sequence prediction problem using LSTMs in a keras architecture. To illustrate the problem I give the following example which resemble the problem I must solve. Lets ...
1
vote
1answer
81 views

Should output data scaling correspond to the activation function's output?

I am building an LSTM with keras which have an activation parameter in the layer. I have read that scaling on the output data ...
1
vote
0answers
24 views

Should I normalize the target data the same way as the input data?

This is the way I'm currently transforming my input data ...
1
vote
1answer
42 views

Normalize data from different groups

I have data that have been grouped in 27 groups by different criteria. The reason for these groupings is to show that each group has a different behavior. However, I would like to normalize everything ...
1
vote
0answers
21 views

How do I discern document structure from differently-tagged XML documents?

I have a body of PDF documents of differing vintage. Our group had exported the documents as text to feed them into a natural-language parser (I think) to pull out subject-verb-predicate triples. ...
1
vote
0answers
39 views

How Google Trends is normalized?

I have a daily series from Google Trends, using the range "today 3-m", but it comes that the last day is not available from this query. For example, today is March,24th and the last day using this ...
1
vote
0answers
61 views

MAPE over 100% after normalization of dataset

I try to forecast power demand for next 24 hours. Years 2017 and 2018 are my training set, 2019 is test set. I use multistep vanilla LSTM . In first step I used original data with any preparation and ...
1
vote
0answers
43 views

De-Normalizing Predicted Values from Neural Network

When fitting the neural network model I normalized the data using the min/max formula: normalized = (data - min(data)) / (max(data) - min(data)) I fitted the ...
1
vote
0answers
18 views

Dynamic time warp z score normalisation not working

I have data that looks like this A bit of background, these are soil moisture graphs of different depths. I wish to investigate how long it takes for water to drip down from one depth to another, ...
1
vote
0answers
14 views

should I normalise data of disparate frequencies

The dataset consists of rows of time series data. Most of the time data remains constant and only updates when there are significant changes. Each time series looks something like this where each ...
1
vote
0answers
64 views

BatchNorm vs InstNorm from the perspective of feature distributions

What I understand so far... The main purpose of BatchNorm is to overcome covariance shift -- more specifically what the authors of BatchNorm coined "internal covariance shift". Covariance shift is ...
1
vote
1answer
104 views

Correcting for one of multiple strong batch effects in a dataset

I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it ...
1
vote
0answers
17 views

QuantileTransform before or after long-to-wide format transformation?

I perform the following steps: Perform quantile transform Transform it from wide to long format Train the anomaly detection model on the wide-format data Will there be some significant change if I, ...
1
vote
0answers
31 views

Normalising energy data using weather data?

I'm currently doing some research into energy use, and normalising it is critical for comparison across years. I'm considering ways to normalise the daily data (kWh) so that I can run statistical ...
1
vote
0answers
62 views

How can detect and highlight outliers by using gaussian function and normalize the data elegantly?

I tried to normalize the data by using Gaussian function 2 times on both positive and negative numbers of each parameter of this dataset. The dataset includes missing data as well. The problem is I ...
1
vote
1answer
113 views

Data normalization of count data for neural networks

I have a sparse matrix of count data that I'm using as input to a neural network. I know, usually, the input data should be normalized (e.g. via min-max scaling, $z$-score standardization, etc.). But ...
1
vote
2answers
708 views

Time series normalization using min max technique

I have a time series dataset and I would like to normalize the data (diff which is of type list) as below using Min Max ...
1
vote
0answers
14 views

Metrics for stain normalization

Are there any metrics or methods for assessing stain normalization techniques?
1
vote
0answers
42 views

Why is a normalized dependent variable important for model?

If we have a highly skewed dependent variable, is it good practice to remove the outliers to force the data into a more normalized shape? If so, why is this important?
1
vote
0answers
8 views

Word frequencies in unbalanced case-control dataset

I have a case-control cohort for which I'm doing analysis of clinical notes. The ratio of cases to controls is 1:4. What I'm looking at is the relative frequency of certain words (e.g. overdose, ...
1
vote
0answers
132 views

Regression Decision Tree - Normalize or Split into Ranges a continuos feature

I have in my dataset a feature named distances which ranges goes from 200 to 12000 (more or less). Since the other features have got values under 50 I need to do ...
1
vote
0answers
57 views

How to preserve levels with Z-score?

I normalize time series data using a sliding window to be fed into a Neural Net. I normalize each window independently using Z-score. (Because on the entire dataset proven to be useless. Because on ...
1
vote
0answers
38 views

How to pre-process frequency of a series of signals?

I use a neural net to generate predictions based on a time series of signals. I use a sliding window to feed the data to an LSTM model. The input signals have a random frequency that - I believe - is ...
1
vote
1answer
226 views

Should Principal components be normalized before applying K means on them?

I want to get the Principal components of a dataset and apply K mean clustering on them. Do I need to Normalized the PCA output before applying Kmeans on them ?