Questions tagged [normalization]
Normalization is a technique often applied as part of data preparation for machine learning. The goal of normalization is to change the values of numeric columns in the dataset to use a common scale, without distorting differences in the ranges of values or losing information.
295
questions
0
votes
0
answers
16
views
TFRobertaSequenceClassification for Address Normalization task
I have dataset with two column: one with faulty addresses, and other with correct addresses. I want to train a model such that, I can use it later for correcting all the incoming faulty addresses.
I ...
0
votes
0
answers
24
views
Data Preprocessing in the Wild?
I am new to ML, NN, and data science as a whole so the following question might sound silly. How can we perform inference when the model is deployed in the wild?
To my understanding, cleaning/...
1
vote
2
answers
57
views
weighting voting classifier (MAE and MSE)
I am trying to optimize the weights of a Voting Regressor problem. To achieve the best score, I am considering both MAE and MSE as parameters, using the following formula:
score = w * MAE + (w-1) * ...
0
votes
0
answers
14
views
Comparing multiple multivariate datasets
Take the two datasets below:
default rate
state
age
income
asofdate
10
Texas
55
100,000
202309
14
Texas
35
97,000
202309
18
Texas
55
95,000
202308
22
Texas
35
95,000
202308
8
New York
21
55,000
...
- 1
0
votes
0
answers
10
views
How to account for exponential oversampling?
I have a dataset of frequencies from 20Hz to 20kHz. The measurement rig that created the dataset doesn't sample evenly across this frequency range. There is a much higher density of sampling happening ...
- 101
0
votes
0
answers
16
views
Riemannian metric in Layer Normalization
I'm reading a paper about Layer normalization, and I couldn't find any clear explanation for this part:
Q1. Can anyone describe the derivation of the first equation in (8)?
Q2. I cannot understand ...
0
votes
1
answer
66
views
Histogram of Oriented Gradients (HOG) - Why normalize 16x16 blocks and not the whole picture?
I'm trying to learn Histogram of Oriented Gradients (HOG)
I understand why we compute the gradient and the orientation and also map every gradient into a 9 binaries histogram that spans from 0 to 180.
...
- 115
0
votes
2
answers
144
views
What information is lost due to normalization?
Normalization is usually seen as a good thing for data preprocessing before training a model.
But I am wondering if there are some information in the data that might be lost during this process.
The ...
- 119
0
votes
1
answer
100
views
How would normalizing be affected by outliers? And how to avoid it?
I have a data set that boils down to Three clomuns: 1.Supplier name 2. Number of transactions with supplier 3. Total value of those transaction.
I'm trying to find the best way to rank all suppliers ...
- 1
0
votes
1
answer
31
views
Converting categorical to the percentage
How do I convert the categorical value to the percentage?|
I have this asset wellness data:
Poor: 3
Warning: 27
Good: 120
How do I convert it to the percentage ...
1
vote
1
answer
34
views
Should I normalize entire dataset or normalize every x_train and every y_train for timeseries forecasting LSTM?
Suppose I have list like this:
...
2
votes
2
answers
176
views
Why are normal distributions so important in deep learning?
I am currently reading on normalization/standardization techniques as well as batch normalization in deep learning and I don't really understand why normal distributions are so important inside deep ...
0
votes
1
answer
153
views
Why apply min-max normalization to each individual mel spectrogram for a training set?
I am watching a tutorial on using mel spectrograms to classify the audio's genre via CNN. My question is why apply local min-max normalization to each individual mel spectrogram? What I mean by local ...
0
votes
0
answers
13
views
Scaling datasets for multi-dataset time series
Suppose that I have training data with dimension $(N,H,F)$, where $N$ represents the number of different datasets, $H$ is the history size and $F$ is the input size. Normalizing each dataset over the ...
- 137
0
votes
0
answers
22
views
Should I specify 'reference=train_set' when creating a evaluating lightgbm.Dataset?
According to the docs, we should offer a argument reference=train_set when creating a validating lightgbm.Dataset.
I know a little about the reason, i.e. there are ...
- 101
1
vote
0
answers
16
views
How to normalize(or other) the audio data so that the same labels with the similar characteristics from different records?
I am trying to detect swallows from recordings taken from hospital. I manually labelled the recordings on the Praat. Now the valid labels are silence, swallows and nonswallows(noise, enviromenment ...
- 11
0
votes
0
answers
65
views
Classical Perceptron Algorithm taking too long to evaluate on un-normalized data
I need to implement classical perceptron algorithm from scratch using numpy and pandas for an assignments.
I have done so using this algorithm:
I have a linearly seperable dataset of 568 rows and 30 ...
- 1
1
vote
0
answers
51
views
NANs, Infinities, and very large losses with normalizing flows
I am new to normalizing flows and have been trying to use them with a high-dimensional dataset, and I have been running into very large numbers and errors with sampling that don't occur when I use a ...
- 31
0
votes
2
answers
27
views
How will a model handle real-life values in real-life applications without scaling?
I am learning ML and facing confusion about data scaling. For example, I have the following data:
Weight(KG)
Balance($)
75
3401542
99
4214514
Now, if I use StandardScaler, I may get something like ...
0
votes
0
answers
27
views
Scaling/Normalization again
I would like to ask for normalization .. again.
I'm working on LSTM model having a 100x4 dataset used for timeseries prediction.
My question is on which set of data the scalar/normalizer should be run ...
- 101
0
votes
1
answer
44
views
Normalization / Overfitting Issues
I have a dataset with 608 inputs and I'm trying to output a single 1 or 0 result. My validation data has 69.12% 0's. When ran, my model always returns 69.12% accuracy, presumably because it's "...
- 3
1
vote
1
answer
61
views
Dictionary-based text analysis- dealing with length
I am working on an analysis using a dictionary-based text-as-data approach. I have a dataset of texts (n=1200), and I am applying a dictionary of 50 words (I tokenize the text with each word being one ...
0
votes
1
answer
42
views
Python dataset normalization for convolutional autoencoder
I have a csv files which contain pixel neighboorhood information. Here an example of the dataset:
...
- 137
0
votes
0
answers
84
views
Do you need to normalize labels for models other than neural nets?
As mentioned here, normalizing the target variable often helps a neural network converge faster.
Does it help in convergence, or is there otherwise a reason to use it, for any type of model other than ...
0
votes
1
answer
59
views
Normalising data for simple linear regression
Consider a simple linear regression problem where:
X = [1,2,3,4,5,100,200]
Y= [2,4,6,8,10,200,400]
Clearly, the relationship is of the form $y=2x$; While trying ...
- 13
3
votes
1
answer
772
views
Out of range [0,1] MinMaxScaler for test data
I know that for MinMaxScaler we should apply it to train data, then apply it, with the obtained parameters, over test data:
...
- 31
0
votes
0
answers
30
views
What's the difference between transforming data (Square-root/log/Square/Cube) and adding Polynomial Terms for better fitting a regression line?
The immediate difference in both the approaches might seem that when we are introducing Polynomial features for Polynomial regression we are also including the original term in our linear equation. ...
0
votes
0
answers
24
views
Is it sensible to use conventional scaling methods to scale features that have meaningful (+/-)sign?
I'm working on a project where the input features contain velocities of objects. The sign in the velocity feature is meaningful i.e. negative velocity means an object moving to the left while positive ...
- 101
0
votes
0
answers
137
views
How to use MinMaxScaler when X_train and X_test are different sizes
I'm using an LSTM through Keras to predict a time series. My inputs are the previous measurements of one time series(v1) at time 0-59 seconds and the goal is to predict the measurement of a different ...
0
votes
0
answers
26
views
What is the NLP task that convert "your" to "one's", "is" to "be"?
In lemmatization, "is" remains the same; in stemming, 'your' remains the same; text normalization is irrelevant.
It can be solved using rules(mapping, like "my" -> "one's&...
- 516
0
votes
1
answer
55
views
Normalising Image Data
Hi I am wondering when it comes to normalising images across each of the channels, do you use the same scaling factors that is used for training for testing set as well or separate ones.
In ...
- 103
0
votes
1
answer
201
views
Standardization after log transformation
I have a few question about log transformation and standardization.
First: Should I standardize my features after doing log transformation?
Second: I still do not understand, because when doing log ...
- 138
1
vote
1
answer
57
views
How to do an incremental update for the mean and standard deviation of tensor data?
I have a big dataset (some 400Gb) consisting of tensor data (shape is $(600, 600, 10)$) and I want to normalize this dataset before feeding it to a neural network but this dataset can't fit in my ...
- 13
0
votes
2
answers
37
views
Normalization of predictors
Does normalization (ex: log or sqrt) of skewed features help to reduce model's bias? or it is better to leave the skewed distribution of predictors as it is?
- 161
0
votes
1
answer
32
views
How "normal" should my input data be?
When training a neural net, I understand the value in normalising the input data to have mean = 0 and stdev = 1 (standardising the data).
But I often see people make the data even more "normal&...
3
votes
1
answer
89
views
What justifies feature scaling?
Although I can understand the significance of feature scaling in some cases (e.g. when gradient descent is involved), I don't feel I understand the necessity of this process in general. But there a ...
- 181
0
votes
1
answer
73
views
why we have problem with gradients when feature values are of different range?
A blog below mentioned.
" Because different features do not have similar ranges of values, gradients may take a long time, oscillate back and forth, and take a long time before they can finally ...
- 11
0
votes
1
answer
95
views
How to convert city name prefix abbreviations?
Is there any standard tool or library or list for expanding town name abbreviations? For example "MT HOLLY" -> "MOUNT HOLLY" or "ST MICHAELS" -> "SAINT ...
- 121
2
votes
0
answers
30
views
When is scaling and centering important?
There are some models such as PCA or SVM where scaling and centering of training data is essential.
There are some models, mostly tree-based where scaling and centering is not required at all.
I don't ...
- 121
0
votes
0
answers
134
views
Why input normalization leads to worse performance?
I'm building quite simple NN with Dense layers followed by ReLU activations and I noticed something unexpected. Generally, I've been confident that normalizing the input to have mean of 0 and standard ...
- 182
0
votes
0
answers
123
views
Do I need to encode numerical variables like "year"?
I have a simple time-series dataset. it has a date-time feature column.
...
- 1
0
votes
2
answers
94
views
Standardization in combination with scaling
Would it be ok to standardize all the features that exhibit normal distribution (with StandardScaler) and then re-scale all the features in the range 0-1 (with <...
- 119
0
votes
1
answer
29
views
Normalizing data from same variable but different individuals
I'm new to machine learning. I have the following scenario:
I have five individuals that are each carrying an accelerometer. That sensor measures movement/acceleration on a scale from 0 to 255, 0 ...
- 101
1
vote
0
answers
45
views
Normalization and Denormalization
I have few queries. 1) Is normalization required for ANN / CNN /LSTM ? 2) If we normalize the data with MinMax Scaler, then in that case how to denormalize it and when to denormalize it so that we ...
- 11
0
votes
0
answers
182
views
How to save pixels after normalization
I want to normalize my images and use them in the training. But I couldn't find a way to save images after making changes below...How can I save it?
...
- 1
0
votes
2
answers
126
views
Is normalization needed for TargetEncoded Variables?
Basically the title.
If I encode the address of people (the cities they live in) with a target encoder, do I still need to normalize that column? Of course, the capital is going to have more citizens ...
- 1,310
1
vote
1
answer
82
views
How To Normalize Feature That Depends Arbitrarily On Date Published
Working on an ML project to predict the number of listens a certain podcast episode of my podcast will get in the first 28 days.
The problem is that when I first started recording the podcast would ...
- 113
0
votes
1
answer
171
views
Normalize new data and old model
I'm playing with machine learning and LSTM. My goal is to learn something new and work with real data.
Currently, I'm trying to predict bitcoin price.
I have understood the necessity to normalize data,...
- 101
0
votes
0
answers
35
views
Impact of log transformation and Normalisation in the context of EDA and ML
Is data normalisation an alternative for log transformation? I understand that both helps us to normalisation helps me to make my distribution gaussian.
Thanks in advance for your help!
- 1
0
votes
0
answers
2k
views
Normalizing images with OpenCV (divide by 255)
I'm loading images from my dataset, which are all of resolution 200x200 and in RGB format.
I'm loading them using OpenCV for Python, with the following code:
...
- 1