Questions tagged [normalization]

The tag has no usage guidance, but it has a tag wiki.

Filter by
Sorted by
Tagged with
1
vote
1answer
21 views

Multi-Output Regression

I have got an .xlsx Excel file with an input an 2 output columns. And there are some coordinates and outputs in that file such as: x= 10 y1=15 y2=20 x= 20 y1=14 y2=22 ... I am trying to do that ...
1
vote
1answer
29 views

Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance?

I am using raw data set with 4 feature variables to do a Binominal Classification using Logistic Regression Algorithm. I made sure that the class counts are balanced. i.e., an equal number of ...
3
votes
3answers
59 views

Why models performs better If normalize test data and train data separately?

Many threads (and courses) such as this and this one suggest that you should apply normalization to the test data using the parameters used in the training set. But other some discussions I've found ...
2
votes
0answers
18 views

Comparing feature importance in LightGBM + Scikit

I have a model trained using LightGBM (LGBMRegressor), in Python, with scikit-learn. On a weekly basis the model in re-trained, and an updated set of chosen features and associated ...
0
votes
4answers
65 views

What is the purpose of standardization in machine learning?

I'm just getting started with learning about K-nearest neighbor and am having a hard time understanding why standardization is required. Reading through, I came across a section saying When ...
2
votes
1answer
21 views

How to normalize a position in 3d game?

I'm pretty new to anything and everything related to this kind of stuff, I was wondering how would I normalize the coordinates of a entity in my game for a nerual network? Would it just be the same ...
0
votes
0answers
16 views

Is it necessary to perform Z-score or Min-Max normalization on L1-normalized data?

I have a dataset which contains vectors that generated from subtitles and have been L1 normalised, I want to calculate cosine similarity & Euclidean distance, I thought it is better if I use Z-...
0
votes
0answers
11 views

QuantileTransform before or after long-to-wide format transformation?

I perform the following steps: Perform quantile transform Transform it from wide to long format Train the anomaly detection model on the wide-format data Will there be some significant change if I, ...
0
votes
0answers
20 views

How can I reduce the noise of prediction graph?

I am trying to use LSTM to predict a time series data as you can see in the following image, the predicted graphs is very noisy: The original data is looking like this: That I normalized it like ...
0
votes
2answers
39 views

Should we denormalize our data after normalization?

If we use sklearn library's preprocessing.normalie() function to normalize our data before learning, like this: ...
0
votes
0answers
10 views

Normalising energy data using weather data?

I'm currently doing some research into energy use, and normalising it is critical for comparison across years. I'm considering ways to normalise the daily data (kWh) so that I can run statistical ...
0
votes
0answers
12 views

What is the difference between row-wise and column-wise Z-score normalization?

I have a data set, each row represents a movie name, each column is a feature (such as genres), I want to perform cosine similarity to find out the similarity between each movie, before that I need to ...
1
vote
0answers
12 views

How can detect and highlight outliers by using gaussian function and normalize the data elegantly?

I tried to normalize the data by using Gaussian function 2 times on both positive and negative numbers of each parameter of this dataset. The dataset includes missing data as well. The problem is I ...
0
votes
0answers
12 views

How to normalize old accumulated ratings

I am playing around with data-set which contains movies and their ratings by various users. I am trying to rank these movies based on their user ratings. However, as obvious, many of the old movies (...
1
vote
1answer
76 views

How to standardize my data (Univariate Time Series Forecasting using Keras LSTM)?

Let be $X = (X_1,...., X_n)$ an univariate time serie. I would like to know how to standardize my data when I split it into train and test data. Let me explain you how I tranform $X$ so that I can fit ...
1
vote
0answers
24 views

Does feature normalization improve performance of Hidden Markov Models?

For training a Hidden Markov Model (HMM) on a multivariate, continuous time series, is it preferable to scale the data somehow? Some pre-processing steps may be: Normalize to 0-mean and unit-variance ...
6
votes
3answers
580 views

Data normalization before or after train-test split?

Which one is the right approach to make data normalization - before or after train-test split? Normalization before split ...
1
vote
1answer
25 views

How many normalization methods are there and what are they for?

this post lists 5 types of normalization. Zscore MinMax Logistic LogNormal TanH is there any other types of normalization that are most commonly used in machine ...
0
votes
2answers
38 views

Would K-means be Appropriate to Use with Four or More Variables?

Just a general question that I'm trying to mentally visualize. I'm fairly new to using k-means clustering and have used it before on two variables, which creates a 2-D plot of points. I also know, ...
1
vote
1answer
101 views

is there a way to normalize [-3,1] to ${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\sqrt{10}}\\ \end{bmatrix}}$ with python?

I am learning SVD by following this MIT course The lecturer is trying to normalize a vector $${\begin{bmatrix} -3\\ 1\\ \end{bmatrix}}$$ to $${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\...
4
votes
4answers
165 views

Should input images be normalized to -1 to 1 or 0 to 1

Many ML tutorials are normalizing input images to value of -1 to 1 before feeding them to ML model. The ML model is most likely a few conv 2d layers followed by a fully connected layers. Assuming ...
0
votes
1answer
41 views

How to normalize test data according to the training data if the normalization on the training data is performed row wise?

I read on several places about the normalization of features in the machine learning method. But I normalize my training data row-wise as shown in the following ...
0
votes
2answers
48 views

Should we normalize test data by choosing maximum and minimum value of training data?

I'm training my CNN network with one model's data whereas i'm testing it with another model's data. I perform min max normalization on each sample. And every samples ranges [-1,1]. My question is that ...
0
votes
1answer
15 views

How to scale exponential data for a regression problem?

I understand that I should be scaling features between (0, 1) before feeding them into a neural network. However, what happens if future data could be larger than my current training data? For ...
3
votes
0answers
26 views

A Deep CNN model delivering better results with standardization, when compared with normalization

I developed a deep CNN model, based on the architecture discussed in this paper, to generate predictions for time series data. My training data is shown in the figure below: In order to train the ...
6
votes
2answers
487 views

Which comes first? Multiple Imputation, Splitting into train/test, or Standardization/Normalization

I am working on a multi-class classification problem, with ~65 features and ~150K instances. 30% of features are categorical and the rest are numerical (continuous). I understand that standardization ...
2
votes
1answer
86 views

MinMaxScaler when LSTM predictions fall outside of training range?

I am using MinMaxScaler on my training set and applying the transformations to my test set and inverse_transform to my model’s outputs. If this were, say, a stock prediction problem, my training set ...
1
vote
1answer
44 views

Softmax function result for already normalized probabilities

Isn't the aim of softmax function normalizing the probabilities such that they all sum to 1? So when we apply this method to the already normalized numbers, it would change them. what do these new ...
1
vote
1answer
40 views

Scaling features separately?

I have some features which are in the thousands, which I scale to the max values of these. This solves the general scaling problems, as well as preserves an important absolute value relationship ...
1
vote
0answers
48 views

Data normalization of count data for neural networks

I have a sparse matrix of count data that I'm using as input to a neural network. I know, usually, the input data should be normalized (e.g. via min-max scaling, $z$-score standardization, etc.). But ...
0
votes
0answers
11 views

Normalization in SVM classifier

I am trying to normalize my features for a classification model with 3 class outputs. There are two kinds of features. First is medical test results and second is patient information such as age. The ...
0
votes
0answers
54 views

Normalization and probability density function

I am wondering is it reasonable to calculate the probability density function ( or calculate some measures of the probability distribution such as skewness) of data after standardizing the data (or ...
1
vote
2answers
98 views

Time series normalization using min max technique

I have a time series dataset and I would like to normalize the data (diff which is of type list) as below using Min Max ...
1
vote
0answers
24 views

Why is it necessary in batch normalization to multiply and add a parameter to the result?

How do we decide on which layer we want to add batch normalization. So if we have chosen a layer to apply batch norm to then why don't just normalize it why are we multiplying and scaling it by some ...
1
vote
2answers
53 views

How to normalize data from multiple sources?

I am trying to model an individuals' purchasing behavior using different data sources (ex: Zalando, Otto, etc.,). When I combine data sources, I see that the data across these channels is very ...
0
votes
1answer
39 views

Normalization(minmax) gives me worse results than before in KNN, follow up actions?

Hello I'm studying a classification problem with KNN right now. I have many numeric features that I normalized with MinMaxScaler, I also got some OHE categorical features that not seem to cause the ...
2
votes
1answer
139 views

Normalizing the data set

I have two questions : Why doesn't normalization have any effect on linear regressor performance (mathematical approach is appreciated ) ? When we normalize the training set we ought to normalize ...
1
vote
0answers
12 views

Metrics for stain normalization

Are there any metrics or methods for assessing stain normalization techniques?
0
votes
1answer
81 views

Scaling sparse data for PCA

Not sure how I should interpret the scaling. Is it correct to convert the sparse matrix to a dense matrix by padding with 0's and scale normally?
3
votes
0answers
40 views

Standardizing data from different distributions

I have energy data from different homes (~200), and I wish to merge all the energy data to create a combined dataset. Since the individual homes have different power consumption, I get different ...
1
vote
0answers
33 views

Why is a normalized dependent variable important for model?

If we have a highly skewed dependent variable, is it good practice to remove the outliers to force the data into a more normalized shape? If so, why is this important?
8
votes
2answers
3k views

When to use Standard Scaler and when Normalizer?

I understand what Standard Scalar does and what Normalizer does, per the scikit documentation: Normalizer, Standard Scaler. I know when Standard Scaler is applied. But in which scenario is Normalizer ...
0
votes
0answers
8 views

Rescale error metrics from Keras net to make errors interpretable

I am using a Keras LSTM to predict a continuous output in a time series data set. Before I train, I scale the inputs by mean centering the data. I would like my error metric (specifically mean squared ...
4
votes
1answer
35 views

Should one normalize the frequency values when feeding it as an input to machine learning model?

Consider an unsupervised data. The data is in the form of a csv file( I am using pandas dataframe for this). Its a web page data at different time steps and the way I am converting data to be fed to ...
2
votes
1answer
105 views

Why does test data need to be normalized on train data mean and std?

I understand why it is usefull to normalize data in general (at least I think I do). You take the mean and the standard deviation of the train data and apply it to both, the train and test data. Why ...
0
votes
0answers
159 views

Normalizing Jaccard similarity scores in relation to differences in document length

The Jaccard similarity of two documents A and B can be defined as the size of their intersection (how many tokens are in both docs) divided by the size of their union (total number of tokens found in ...
1
vote
1answer
181 views

Standard Scaler drops accuracy significantly in Scala Spark

I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly. Without the accuracy is ~90% (on training ...
2
votes
1answer
23 views

Image normalisation methods

I have found some research papers specifying explicitly the normalisation technique they used to get the results. What difference do IMG /255.0 And ...
0
votes
2answers
87 views

Why normalization kills my accuracy

I have a binary car sound classifier. I have a feature set that is extracted from audio with size of 48. I have a model(multi layer neural network) that has around %90 accuracy on test and validation ...
0
votes
2answers
1k views

How to normalize just one feature by scikit-learn?

Wanna apply a specific scaler, say StandardScaler, on a specific feature, keeping other features intact. the dataset format is something like: [ [1, 0.2, 1000], [2, 0.1, 2400], [3, 0.9, 7620] ] I ...