Questions tagged [normalization]

The tag has no usage guidance, but it has a tag wiki.

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

How do I discern document structure from differently-tagged XML documents?

I have a body of PDF documents of differing vintage. Our group had exported the documents as text to feed them into a natural-language parser (I think) to pull out subject-verb-predicate triples. ...
0
votes
0answers
5 views

The need to normalize input data for DNN when using He_initialization

I'm reading Geron's "Hands-On Machine Learning with Scikit_learn, Keras & Tensorflow" I noticed in the problem set after chapter 11, he ran a DNN using ELU activation and He Initialization. But ...
0
votes
0answers
13 views

When normalizing unstructured data, can you take the mean and standard deviation globally?

I'm watching this FastAI machine learning lesson, which is about the famous MNIST dataset. I've gottent to the point where we should normalize the data and, much to my surprise, the instructor uses ...
5
votes
3answers
53 views

How to normalize data without knowing the min and max values?

I have a Lending club dataset from Kaggle; it contains many different columns: there are for example dummy variables, years, amount of the loan...ect I want to normalize the data in the training and ...
1
vote
0answers
16 views

How Google Trends is normalized?

I have a daily series from Google Trends, using the range "today 3-m", but it comes that the last day is not available from this query. For example, today is March,24th and the last day using this ...
0
votes
0answers
33 views

How to correctly manage predictions when the inputs are outbound the original scaling range?

I have a neural network for a regression problem that was trained using MinMaxScaler(0,1) for features and I have two questions with this. I often find that scaling the output (or target variable) ...
0
votes
0answers
12 views

MAPE over 100% after normalization of dataset

I try to forecast power demand for next 24 hours. Years 2017 and 2018 are my training set, 2019 is test set. I use multistep vanilla LSTM . In first step I used original data with any preparation and ...
0
votes
1answer
15 views

Why normalize function has a different result on a matrix vs single value?

I have a matrix like: B=[ 1.5035; 1.5728; 1.6485; 1.5369; 1.5467; 1.572; 1.5374; 1.787; 1.5825; 1.6905]; Using normalize function like normalize(B,'range') ...
0
votes
0answers
18 views

Index confusion after splitting, normalizing and oversampling the data set

I am working on a data set with 16,259 samples with class label 0 and 1,639 samples with class label 1. In order to balance the data set, I am applying oversampling (after normalization). The problem ...
0
votes
0answers
11 views

Can I use batch normalization and layer normalization both to CNN-RNN model?

I'm performing a classification task with time series data. Therefore, I designed an 1DCNN-LSTM model. Currently, 1d-batch normalization layers are applied for CNN part, but I'm not sure to use layer ...
0
votes
0answers
14 views

Is there a way to normalize a similarity matrix by row and column in a way such that only one entry per row or column is approximately 1

I am computing similarities between 2 vectors. My goal is to have approximately 1 matching sample with similarity ~1, for each sample, without having any samples that are similar to many other samples....
1
vote
0answers
21 views

De-Normalizing Predicted Values from Neural Network

When fitting the neural network model I normalized the data using the min/max formula: normalized = (data - min(data)) / (max(data) - min(data)) I fitted the ...
1
vote
0answers
12 views

Dynamic time warp z score normalisation not working

I have data that looks like this A bit of background, these are soil moisture graphs of different depths. I wish to investigate how long it takes for water to drip down from one depth to another, ...
1
vote
1answer
19 views

Dealing with issues in “test” predictons for single “items” (null values, standardization in place, etc)

I know this is kind of a broad question but I have tried to scour both this forum and the internet in general to no avail for this particular situation. So imagine I have a model trained for which, ...
0
votes
1answer
25 views

How can I train my CNN to learn (numerically) smaller values better?

I'm using a CNN to model a problem that involves precise numerical values from a physical simulation. After months of design/redesign and optimization, I've noticed that the majority of the "error" ...
2
votes
1answer
30 views

Does it make sense to normalize?

My data consists of multiple NHL seasons where the total number of goals scored by each player is kept track. I intend to compare the distribution of goals scored between each season. My thinking ...
2
votes
1answer
32 views

Batch Normalization vs Other Normalization Techniques

In the context of neural networks, I understand that batch normalization ensures that activation at each layer of the neural net does not 'blow-up' and cause a bias in the network. However, I don't ...
2
votes
1answer
41 views

Differences between normalization and standarization in multiple regression

Consider the following question regarding multiple regression 1) Can someone explain why we have to transform dependent variable using log-transformation (Normalization) when appear positive skewed y ...
1
vote
3answers
82 views

why to use Scaler.fit only on x_train and not on x_test for normalizing value using MinMaxScaler?

while normalising the data everone is saying that we need to fit only on x_train and not on x_test ? why is that we should not fit x_test ? if we should not fit the scaler on x_test then why we need ...
1
vote
1answer
37 views

How to append new numerical features to an embedding from word2vec, such that KNN on embeddings is not biased for one feature?

I am working on similarity calculation between entities of similar type. For each entity I am able to make a vector that comprises of multiple vectors itself. A = 50*1 vector B = 100*1 vector C = 50*...
0
votes
1answer
58 views

CrossMapLRN2d in pytorch

I had to convert a code written in pytorch to keras (with tensorflow backend). But there was this layer called CrossMapLRN2d which had no direct counterpart in Keras. So wanted to know what does this ...
1
vote
2answers
75 views

why does making the target variable normally distributed helps?

while working on some regression problems I have found that if the target variable is skewed, making it normally distributed(using transformations) almost always helps. Why is that? Should we also ...
2
votes
1answer
42 views

Normalization in production

I am currently writing a machine learning pipeline for my time series application. At the end of each month, I get the data gathered, normalize it ([0, 1]), retrain the ML model with the new ...
0
votes
0answers
24 views

How to do normalize distribution technique on a simple matrix?

This is just a practice about normal distribution in MATLAB and not a real project. I created a simple matrix and want to test normal distribution on it, it's ...
0
votes
1answer
19 views

How to set and check normal distribution on a data set?

Sorry if my question is simple, I have a data set with two class and want to check and set normal distribution on it(if it was necessary, in MATLAB). But the question is that I should use it on every ...
1
vote
0answers
12 views

should I normalise data of disparate frequencies

The dataset consists of rows of time series data. Most of the time data remains constant and only updates when there are significant changes. Each time series looks something like this where each ...
0
votes
1answer
37 views

How to scale a variable when not knowing the maximum

I have a dataset with different features where some of them are not categorical, so they need to be scaled or normalized (especially the target). However, normalizing between 0-1 for instance means ...
0
votes
2answers
43 views

Multi-class classification with mostly zero valued data

I implemented a multi-class regression and wanted to test it using the MNIST dataset. I realized that if I use standardization $X \leftarrow \frac{X-mean(X)}{std(X)}$, over 50% of all features ...
0
votes
0answers
75 views

What is “style normalization” referred to in the Adaptive Instance Normalization paper?

In "Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization", the authors argue that the significant performance boost from instance normalization is not only due to contrast ...
0
votes
0answers
22 views

BatchNorm vs InstNorm from the perspective of feature distributions

What I understand so far... The main purpose of BatchNorm is to overcome covariance shift -- more specifically what the authors of BatchNorm coined "internal covariance shift". Covariance shift is ...
0
votes
1answer
36 views

PCA and orange software

I am analysing if 15 books can be grouped according to 6 variables (of the 15 books, 2 are written by an author, 6 by an other one, and 7 by an other one). I counted the number of occurrences of the ...
1
vote
1answer
22 views

Normalisation results in R^2 score of 0 - Lasso regression

I am running a regression analysis on a 7000 row dataset with a train/test split of 70%/30%. I am using one variable X to predict a variable ...
0
votes
0answers
48 views

normalization/standardization of input/output of autoencoder and Gaussian Process

I have two machine learning algorithms that deal with time series data. My data consist of 1500 time series, each of 500 time components. The first machine learning algorithm is an autoencoder, ...
0
votes
1answer
9 views

SVM, which range to use when normalizing

I am using the SVM classifier from Scikit Learn. I was wondering is there is a know-best-practice when it comes to normalization. I'm using different normalization tecniques, but all my normalized ...
0
votes
1answer
90 views

Correcting for one of multiple strong batch effects in a dataset

I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it ...
1
vote
1answer
24 views

When would not normalizing input values have higher accuracy?

Right now I'm training a deep neural network for a binary classification problem, with a feature set of winrates. As such, each winrate is bigger or equal to 0 but smaller than 100. I've been getting ...
1
vote
1answer
151 views

can I use z-score normalization even if it doesn't make sense for my data to be negative?

I'm planning to use z-score as a Normalization Method for a Project but I noticed if I do that then I ll have a data in the range [-1, 1] which is wierd because I have data that doesn't make sense for ...
0
votes
1answer
108 views

What is the use of fit method in sklearn.preprocessing.Normalizer()?

According to the documentation of fit(self, X[, y]) method of sklearn.preprocessing.Normalizer(), it does nothing and return the estimator unchanged. I understand that if I intend to normalize data I ...
0
votes
0answers
9 views

Log or StandardScaler for 0.0:1.0 values

If I wanted to normalize a bunch of values that were between zero and 1, where some values were dangerously close to zero such as 0.00001 and others were magnitudes larger like 0.8 - would I take the ...
0
votes
0answers
14 views

Is this interpretation of spectral normalisation mathematically correct?

Hello everyone, this is my first post. I was thinking about the mathematical interpretation for spectral normalization in neural networks the other day, and I came up with an explanation that feels ...
0
votes
1answer
608 views

Is it necessary to normalise data for XGBoost?

MinMaxScaler in scikit_learn is used for data normalization (a.k.a feature scaling). Data normalisation is not necessary for decision trees. Since XGBoost is based on decision trees, is it necessary ...
0
votes
1answer
407 views

If I have negative and positive numbers for a feature, should MinMaxScaler be -1 to 1?

I have a variable X with values ranging from -150 to 400. All the other variables in my training set are positive so I normalized them to be from 0 to 1, or they’re already binary, or they had a ...
0
votes
0answers
19 views

Normalization of Distribution of numeric features and target

Here, I am talking about the Normalization of Distribution of Features/Target, not normalization/scaling of them. I've been seeing Notebooks which normalize some features and the target variable, ...
0
votes
2answers
61 views

How to normalize a data set of multiple time series?

I have the a data set representing the electricity consumption of 25 000 customer. The electricity readings are taken from each smart meter each 15 min for a period of 3 days. The data is takes from ...
2
votes
1answer
48 views

Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance?

I am using raw data set with 4 feature variables to do a Binominal Classification using Logistic Regression Algorithm. I made sure that the class counts are balanced. i.e., an equal number of ...
3
votes
3answers
152 views

Why models performs better If normalize test data and train data separately?

Many threads (and courses) such as this and this one suggest that you should apply normalization to the test data using the parameters used in the training set. But other some discussions I've found ...
2
votes
0answers
69 views

Comparing feature importance in LightGBM + Scikit

I have a model trained using LightGBM (LGBMRegressor), in Python, with scikit-learn. On a weekly basis the model in re-trained, and an updated set of chosen features and associated ...
0
votes
4answers
193 views

What is the purpose of standardization in machine learning?

I'm just getting started with learning about K-nearest neighbor and am having a hard time understanding why standardization is required. Reading through, I came across a section saying When ...
2
votes
1answer
25 views

How to normalize a position in 3d game?

I'm pretty new to anything and everything related to this kind of stuff, I was wondering how would I normalize the coordinates of a entity in my game for a nerual network? Would it just be the same ...
0
votes
0answers
28 views

Is it necessary to perform Z-score or Min-Max normalization on L1-normalized data?

I have a dataset which contains vectors that generated from subtitles and have been L1 normalised, I want to calculate cosine similarity & Euclidean distance, I thought it is better if I use Z-...