Questions tagged [normalization]

The tag has no usage guidance, but it has a tag wiki.

Filter by
Sorted by
Tagged with
0
votes
1answer
37 views

How to standardize my data (Univariate Time Series Forecasting using Keras LSTM)?

Let be $X = (X_1,...., X_n)$ an univariate time serie. I would like to know how to standardize my data when I split it into train and test data. Let me explain you how I tranform $X$ so that I can fit ...
1
vote
0answers
18 views

Does feature normalization improve performance of Hidden Markov Models?

For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be: Normalize to 0-mean and unit-variance ...
4
votes
3answers
419 views

Data normalization before or after train-test split?

Which one is the right approach to make data normalization - before or after train-test split? Normalization before split ...
1
vote
1answer
24 views

How many normalization methods are there and what are they for?

this post lists 5 types of normalization. Zscore MinMax Logistic LogNormal TanH is there any other types of normalization that are most commonly used in machine ...
0
votes
2answers
30 views

Would K-means be Appropriate to Use with Four or More Variables?

Just a general question that I'm trying to mentally visualize. I'm fairly new to using k-means clustering and have used it before on two variables, which creates a 2-D plot of points. I also know, ...
1
vote
1answer
87 views

is there a way to normalize [-3,1] to ${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\sqrt{10}}\\ \end{bmatrix}}$ with python?

I am learning SVD by following this MIT course The lecturer is trying to normalize a vector $${\begin{bmatrix} -3\\ 1\\ \end{bmatrix}}$$ to $${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\...
4
votes
4answers
96 views

Should input images be normalized to -1 to 1 or 0 to 1

Many ML tutorials are normalizing input images to value of -1 to 1 before feeding them to ML model. The ML model is most likely a few conv 2d layers followed by a fully connected layers. Assuming ...
0
votes
1answer
30 views

How to normalize test data according to the training data if the normalization on the training data is performed row wise?

I read on several places about the normalization of features in the machine learning method. But I normalize my training data row-wise as shown in the following ...
0
votes
2answers
33 views

Should we normalize test data by choosing maximum and minimum value of training data?

I'm training my CNN network with one model's data whereas i'm testing it with another model's data. I perform min max normalization on each sample. And every samples ranges [-1,1]. My question is that ...
0
votes
1answer
11 views

How to scale exponential data for a regression problem?

I understand that I should be scaling features between (0, 1) before feeding them into a neural network. However, what happens if future data could be larger than my current training data? For ...
3
votes
0answers
23 views

A Deep CNN model delivering better results with standardization, when compared with normalization

I developed a deep CNN model, based on the architecture discussed in this paper, to generate predictions for time series data. My training data is shown in the figure below: In order to train the ...
6
votes
2answers
465 views

Which comes first? Multiple Imputation, Splitting into train/test, or Standardization/Normalization

I am working on a multi-class classification problem, with ~65 features and ~150K instances. 30% of features are categorical and the rest are numerical (continuous). I understand that standardization ...
1
vote
1answer
26 views

MinMaxScaler when LSTM predictions fall outside of training range?

I am using MinMaxScaler on my training set and applying the transformations to my test set and inverse_transform to my model’s outputs. If this were, say, a stock prediction problem, my training set ...
1
vote
1answer
33 views

Softmax function result for already normalized probabilities

Isn't the aim of softmax function normalizing the probabilities such that they all sum to 1? So when we apply this method to the already normalized numbers, it would change them. what do these new ...
1
vote
1answer
34 views

Scaling features separately?

I have some features which are in the thousands, which I scale to the max values of these. This solves the general scaling problems, as well as preserves an important absolute value relationship ...
1
vote
0answers
42 views

Data normalization of count data for neural networks

I have a sparse matrix of count data that I'm using as input to a neural network. I know, usually, the input data should be normalized (e.g. via min-max scaling, $z$-score standardization, etc.). But ...
0
votes
0answers
8 views

Normalization in SVM classifier

I am trying to normalize my features for a classification model with 3 class outputs. There are two kinds of features. First is medical test results and second is patient information such as age. The ...
0
votes
0answers
53 views

Normalization and probability density function

I am wondering is it reasonable to calculate the probability density function ( or calculate some measures of the probability distribution such as skewness) of data after standardizing the data (or ...
1
vote
2answers
45 views

Time series normalization using min max technique

I have a time series dataset and I would like to normalize the data (diff which is of type list) as below using Min Max ...
1
vote
0answers
22 views

Why is it necessary in batch normalization to multiply and add a parameter to the result?

How do we decide on which layer we want to add batch normalization. So if we have chosen a layer to apply batch norm to then why don't just normalize it why are we multiplying and scaling it by some ...
1
vote
1answer
33 views

How to normalize data from multiple sources?

I am trying to model an individuals' purchasing behavior using different data sources (ex: Zalando, Otto, etc.,). When I combine data sources, I see that the data across these channels is very ...
0
votes
1answer
36 views

Normalization(minmax) gives me worse results than before in KNN, follow up actions?

Hello I'm studying a classification problem with KNN right now. I have many numeric features that I normalized with MinMaxScaler, I also got some OHE categorical features that not seem to cause the ...
2
votes
1answer
104 views

Normalizing the data set

I have two questions : Why doesn't normalization have any effect on linear regressor performance (mathematical approach is appreciated ) ? When we normalize the training set we ought to normalize ...
1
vote
0answers
12 views

Metrics for stain normalization

Are there any metrics or methods for assessing stain normalization techniques?
0
votes
1answer
49 views

Scaling sparse data for PCA

Not sure how I should interpret the scaling. Is it correct to convert the sparse matrix to a dense matrix by padding with 0's and scale normally?
3
votes
0answers
37 views

Standardizing data from different distributions

I have energy data from different homes (~200), and I wish to merge all the energy data to create a combined dataset. Since the individual homes have different power consumption, I get different ...
1
vote
0answers
29 views

Why is a normalized dependent variable important for model?

If we have a highly skewed dependent variable, is it good practice to remove the outliers to force the data into a more normalized shape? If so, why is this important?
7
votes
2answers
1k views

When to use Standard Scaler and when Normalizer?

I understand what Standard Scalar does and what Normalizer does, per the scikit documentation: Normalizer, Standard Scaler. I know when Standard Scaler is applied. But in which scenario is Normalizer ...
0
votes
0answers
6 views

Rescale error metrics from Keras net to make errors interpretable

I am using a Keras LSTM to predict a continuous output in a time series data set. Before I train, I scale the inputs by mean centering the data. I would like my error metric (specifically mean squared ...
3
votes
1answer
33 views

Should one normalize the frequency values when feeding it as an input to machine learning model?

Consider an unsupervised data. The data is in the form of a csv file( I am using pandas dataframe for this). Its a web page data at different time steps and the way I am converting data to be fed to ...
2
votes
1answer
63 views

Why does test data need to be normalized on train data mean and std?

I understand why it is usefull to normalize data in general (at least I think I do). You take the mean and the standard deviation of the train data and apply it to both, the train and test data. Why ...
0
votes
0answers
107 views

Normalizing Jaccard similarity scores in relation to differences in document length

The Jaccard similarity of two documents A and B can be defined as the size of their intersection (how many tokens are in both docs) divided by the size of their union (total number of tokens found in ...
1
vote
1answer
146 views

Standard Scaler drops accuracy significantly in Scala Spark

I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly. Without the accuracy is ~90% (on training ...
2
votes
1answer
21 views

Image normalisation methods

I have found some research papers specifying explicitly the normalisation technique they used to get the results. What difference do IMG /255.0 And ...
0
votes
2answers
73 views

Why normalization kills my accuracy

I have a binary car sound classifier. I have a feature set that is extracted from audio with size of 48. I have a model(multi layer neural network) that has around %90 accuracy on test and validation ...
0
votes
2answers
625 views

How to normalize just one feature by scikit-learn?

Wanna apply a specific scaler, say StandardScaler, on a specific feature, keeping other features intact. the dataset format is something like: [ [1, 0.2, 1000], [2, 0.1, 2400], [3, 0.9, 7620] ] I ...
2
votes
1answer
105 views

Financial Time Series data normalization

I'm using Keras in R to predict financial time series. It's easy to normalize price, simply compute returns or log returns, usually it's enough. I want to use Goldman Sachs Financial Conditions Index ...
0
votes
1answer
132 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
0
votes
2answers
294 views

Keras y-labels range between 0 and 1 instead of binary?

I have X values and corresponding y-labels, until now I used to round my labels <0.5 to 0 and >0.5 to 1. Is it possible to use values between 0 and 1 for "y train"? Using Keras and Tensorflow. ...
1
vote
1answer
87 views

One Hot Encoding of Age

My task is to predict how many years a person has left to live using an MLP. There is one specific feature I'd like to discuss: current age. Statistically, it's a conditional probability. Example: ...
1
vote
1answer
106 views

SVM model classifying into one class only, after standardization

I'm trying to use SVM in R (e1071 package) to classify samples as normal or tumor. I have two separate data sets - Training (~50 samples, 100 features) and Test (~60 samples). These data sets are ...
1
vote
0answers
8 views

Word frequencies in unbalanced case-control dataset

I have a case-control cohort for which I'm doing analysis of clinical notes. The ratio of cases to controls is 1:4. What I'm looking at is the relative frequency of certain words (e.g. overdose, ...
2
votes
2answers
450 views

How can I augment my image data?

What are the correct and common ways to normalize image for CNN? I used to work with text and it was pretty straightforward. Removing stop words, clean text from noise, tokenization, stemming etc. ...
1
vote
2answers
72 views

normalization of probabilities in predicting a poly-neuron output in neural nets

When predicting a poly-neuron output in neural nets, say, predicting multiple handwritten digits and giving an output neuron vector (0.1,...,0.9,0.1,...), many use sth like softmax (or sth like the ...
0
votes
1answer
95 views

PANDAS Within Category Normalization

I'm want to normalize sales data of multiple point of sales (POS), Products and weeks. The dataframe looks like this: ...
1
vote
1answer
301 views

Is it better to use a MinMax or a Log Return normalization to predict stock price movements?

I am trying to use a LSTM model to predict d+2 and d+3 closing prices. I am not sure whether I should normalize the data with a MixMax scaler (-1,+1) using the log return (P(n)-P(0))/P(0) for each ...
0
votes
1answer
280 views

How to normalize data of a different nature?

I am working a price prediction LTSM model for the stock market. I am using multiple features: Open, Close, High and I would like to add the Volume. The 3 first features are of the same nature but ...
1
vote
2answers
111 views

Why normalize when all features are on the same scale?

So I'm doing the tensorflow tutorial found here: https://www.tensorflow.org/tutorials/keras/basic_classification Basically, my input is a [28x28] matrix (image) that I flatten to a [1x784] vector. ...
3
votes
3answers
820 views

normalizing data and avoiding dividing by zero

I have data that I'm compressing with AutoEncoders (3-layer neural network) and I would like to normalize my data first. I would like to try to use the coded latent vector and feed it into an anomaly ...
1
vote
1answer
50 views

Normalizing / standardizing training and validation data

Say I split my data to 80% training and 20% test/validation and I want to standardize it, I think I'm right in saying I shouldn't standardize across 100% of the data, and then do the split, because ...