Questions tagged [normalization]

Normalization is a technique often applied as part of data preparation for machine learning. The goal of normalization is to change the values of numeric columns in the dataset to use a common scale, without distorting differences in the ranges of values or losing information.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Should I log transform the data if the dataset is small?

Suppose I have a dataset with only about 25 rows. All the relevant columns are all integers. The ranges of them go from 1 million to 100 million, and the distributions are all skewed right. In this ...
DSGuest's user avatar
0 votes
0 answers
16 views

TFRobertaSequenceClassification for Address Normalization task

I have dataset with two column: one with faulty addresses, and other with correct addresses. I want to train a model such that, I can use it later for correcting all the incoming faulty addresses. I ...
learner_account's user avatar
0 votes
0 answers
24 views

Data Preprocessing in the Wild?

I am new to ML, NN, and data science as a whole so the following question might sound silly. How can we perform inference when the model is deployed in the wild? To my understanding, cleaning/...
Abdullah Abdulrahman's user avatar
1 vote
2 answers
62 views

weighting voting classifier (MAE and MSE)

I am trying to optimize the weights of a Voting Regressor problem. To achieve the best score, I am considering both MAE and MSE as parameters, using the following formula: score = w * MAE + (w-1) * ...
Guilherme Raibolt's user avatar
0 votes
0 answers
14 views

Comparing multiple multivariate datasets

Take the two datasets below: default rate state age income asofdate 10 Texas 55 100,000 202309 14 Texas 35 97,000 202309 18 Texas 55 95,000 202308 22 Texas 35 95,000 202308 8 New York 21 55,000 ...
rpiston's user avatar
0 votes
0 answers
10 views

How to account for exponential oversampling?

I have a dataset of frequencies from 20Hz to 20kHz. The measurement rig that created the dataset doesn't sample evenly across this frequency range. There is a much higher density of sampling happening ...
db2791's user avatar
  • 101
0 votes
0 answers
18 views

Riemannian metric in Layer Normalization

I'm reading a paper about Layer normalization, and I couldn't find any clear explanation for this part: Q1. Can anyone describe the derivation of the first equation in (8)? Q2. I cannot understand ...
user154010's user avatar
0 votes
1 answer
89 views

Histogram of Oriented Gradients (HOG) - Why normalize 16x16 blocks and not the whole picture?

I'm trying to learn Histogram of Oriented Gradients (HOG) I understand why we compute the gradient and the orientation and also map every gradient into a 9 binaries histogram that spans from 0 to 180. ...
euraad's user avatar
  • 115
0 votes
2 answers
173 views

What information is lost due to normalization?

Normalization is usually seen as a good thing for data preprocessing before training a model. But I am wondering if there are some information in the data that might be lost during this process. The ...
Fraïssé's user avatar
  • 119
0 votes
1 answer
116 views

How would normalizing be affected by outliers? And how to avoid it?

I have a data set that boils down to Three clomuns: 1.Supplier name 2. Number of transactions with supplier 3. Total value of those transaction. I'm trying to find the best way to rank all suppliers ...
Rakuzan's user avatar
0 votes
1 answer
32 views

Converting categorical to the percentage

How do I convert the categorical value to the percentage?| I have this asset wellness data: Poor: 3 Warning: 27 Good: 120 How do I convert it to the percentage ...
Muhammad Ikhwan Perwira's user avatar
1 vote
1 answer
39 views

Should I normalize entire dataset or normalize every x_train and every y_train for timeseries forecasting LSTM?

Suppose I have list like this: ...
Muhammad Ikhwan Perwira's user avatar
2 votes
2 answers
198 views

Why are normal distributions so important in deep learning?

I am currently reading on normalization/standardization techniques as well as batch normalization in deep learning and I don't really understand why normal distributions are so important inside deep ...
Kiran Manicka's user avatar
0 votes
1 answer
174 views

Why apply min-max normalization to each individual mel spectrogram for a training set?

I am watching a tutorial on using mel spectrograms to classify the audio's genre via CNN. My question is why apply local min-max normalization to each individual mel spectrogram? What I mean by local ...
Hayden LaBrie's user avatar
0 votes
0 answers
13 views

Scaling datasets for multi-dataset time series

Suppose that I have training data with dimension $(N,H,F)$, where $N$ represents the number of different datasets, $H$ is the history size and $F$ is the input size. Normalizing each dataset over the ...
Hadar's user avatar
  • 137
0 votes
0 answers
23 views

Should I specify 'reference=train_set' when creating a evaluating lightgbm.Dataset?

According to the docs, we should offer a argument reference=train_set when creating a validating lightgbm.Dataset. I know a little about the reason, i.e. there are ...
EvilRoach's user avatar
  • 101
1 vote
0 answers
16 views

How to normalize(or other) the audio data so that the same labels with the similar characteristics from different records?

I am trying to detect swallows from recordings taken from hospital. I manually labelled the recordings on the Praat. Now the valid labels are silence, swallows and nonswallows(noise, enviromenment ...
Yalçın Cenik's user avatar
0 votes
0 answers
67 views

Classical Perceptron Algorithm taking too long to evaluate on un-normalized data

I need to implement classical perceptron algorithm from scratch using numpy and pandas for an assignments. I have done so using this algorithm: I have a linearly seperable dataset of 568 rows and 30 ...
Prad's user avatar
  • 1
1 vote
0 answers
55 views

NANs, Infinities, and very large losses with normalizing flows

I am new to normalizing flows and have been trying to use them with a high-dimensional dataset, and I have been running into very large numbers and errors with sampling that don't occur when I use a ...
quail's user avatar
  • 31
0 votes
2 answers
27 views

How will a model handle real-life values in real-life applications without scaling?

I am learning ML and facing confusion about data scaling. For example, I have the following data: Weight(KG) Balance($) 75 3401542 99 4214514 Now, if I use StandardScaler, I may get something like ...
Ishrat Hossain's user avatar
0 votes
0 answers
27 views

Scaling/Normalization again

I would like to ask for normalization .. again. I'm working on LSTM model having a 100x4 dataset used for timeseries prediction. My question is on which set of data the scalar/normalizer should be run ...
Paul Paku's user avatar
  • 101
0 votes
1 answer
45 views

Normalization / Overfitting Issues

I have a dataset with 608 inputs and I'm trying to output a single 1 or 0 result. My validation data has 69.12% 0's. When ran, my model always returns 69.12% accuracy, presumably because it's "...
n3uralio's user avatar
1 vote
1 answer
62 views

Dictionary-based text analysis- dealing with length

I am working on an analysis using a dictionary-based text-as-data approach. I have a dataset of texts (n=1200), and I am applying a dictionary of 50 words (I tokenize the text with each word being one ...
Iamembarassed123's user avatar
0 votes
1 answer
45 views

Python dataset normalization for convolutional autoencoder

I have a csv files which contain pixel neighboorhood information. Here an example of the dataset: ...
user979974's user avatar
0 votes
0 answers
86 views

Do you need to normalize labels for models other than neural nets?

As mentioned here, normalizing the target variable often helps a neural network converge faster. Does it help in convergence, or is there otherwise a reason to use it, for any type of model other than ...
bdavidson's user avatar
0 votes
1 answer
60 views

Normalising data for simple linear regression

Consider a simple linear regression problem where: X = [1,2,3,4,5,100,200] Y= [2,4,6,8,10,200,400] Clearly, the relationship is of the form $y=2x$; While trying ...
butwhy's user avatar
  • 13
3 votes
1 answer
801 views

Out of range [0,1] MinMaxScaler for test data

I know that for MinMaxScaler we should apply it to train data, then apply it, with the obtained parameters, over test data: ...
John mx's user avatar
  • 31
0 votes
0 answers
143 views

How to use MinMaxScaler when X_train and X_test are different sizes

I'm using an LSTM through Keras to predict a time series. My inputs are the previous measurements of one time series(v1) at time 0-59 seconds and the goal is to predict the measurement of a different ...
user142789's user avatar
0 votes
0 answers
27 views

What is the NLP task that convert "your" to "one's", "is" to "be"?

In lemmatization, "is" remains the same; in stemming, 'your' remains the same; text normalization is irrelevant. It can be solved using rules(mapping, like "my" -> "one's&...
Lerner Zhang's user avatar
0 votes
1 answer
57 views

Normalising Image Data

Hi I am wondering when it comes to normalising images across each of the channels, do you use the same scaling factors that is used for training for testing set as well or separate ones. In ...
imantha's user avatar
  • 103
0 votes
1 answer
223 views

Standardization after log transformation

I have a few question about log transformation and standardization. First: Should I standardize my features after doing log transformation? Second: I still do not understand, because when doing log ...
Jovian Aditya's user avatar
1 vote
1 answer
59 views

How to do an incremental update for the mean and standard deviation of tensor data?

I have a big dataset (some 400Gb) consisting of tensor data (shape is $(600, 600, 10)$) and I want to normalize this dataset before feeding it to a neural network but this dataset can't fit in my ...
Toshi Mint's user avatar
0 votes
2 answers
37 views

Normalization of predictors

Does normalization (ex: log or sqrt) of skewed features help to reduce model's bias? or it is better to leave the skewed distribution of predictors as it is?
Kusisi Karem's user avatar
0 votes
1 answer
32 views

How "normal" should my input data be?

When training a neural net, I understand the value in normalising the input data to have mean = 0 and stdev = 1 (standardising the data). But I often see people make the data even more "normal&...
JessicaDavies's user avatar
3 votes
1 answer
90 views

What justifies feature scaling?

Although I can understand the significance of feature scaling in some cases (e.g. when gradient descent is involved), I don't feel I understand the necessity of this process in general. But there a ...
ado sar's user avatar
  • 181
0 votes
1 answer
74 views

why we have problem with gradients when feature values are of different range?

A blog below mentioned. " Because different features do not have similar ranges of values, gradients may take a long time, oscillate back and forth, and take a long time before they can finally ...
smsubham's user avatar
0 votes
1 answer
99 views

How to convert city name prefix abbreviations?

Is there any standard tool or library or list for expanding town name abbreviations? For example "MT HOLLY" -> "MOUNT HOLLY" or "ST MICHAELS" -> "SAINT ...
Cerin's user avatar
  • 121
2 votes
0 answers
30 views

When is scaling and centering important?

There are some models such as PCA or SVM where scaling and centering of training data is essential. There are some models, mostly tree-based where scaling and centering is not required at all. I don't ...
xan's user avatar
  • 121
0 votes
0 answers
137 views

Why input normalization leads to worse performance?

I'm building quite simple NN with Dense layers followed by ReLU activations and I noticed something unexpected. Generally, I've been confident that normalizing the input to have mean of 0 and standard ...
Xaume's user avatar
  • 182
0 votes
0 answers
131 views

Do I need to encode numerical variables like "year"?

I have a simple time-series dataset. it has a date-time feature column. ...
smarks70's user avatar
0 votes
2 answers
101 views

Standardization in combination with scaling

Would it be ok to standardize all the features that exhibit normal distribution (with StandardScaler) and then re-scale all the features in the range 0-1 (with <...
Caterina's user avatar
  • 119
0 votes
1 answer
29 views

Normalizing data from same variable but different individuals

I'm new to machine learning. I have the following scenario: I have five individuals that are each carrying an accelerometer. That sensor measures movement/acceleration on a scale from 0 to 255, 0 ...
Benisburgers's user avatar
1 vote
0 answers
46 views

Normalization and Denormalization

I have few queries. 1) Is normalization required for ANN / CNN /LSTM ? 2) If we normalize the data with MinMax Scaler, then in that case how to denormalize it and when to denormalize it so that we ...
Tarun Sharma's user avatar
0 votes
0 answers
186 views

How to save pixels after normalization

I want to normalize my images and use them in the training. But I couldn't find a way to save images after making changes below...How can I save it? ...
Zehra N.'s user avatar
0 votes
2 answers
134 views

Is normalization needed for TargetEncoded Variables?

Basically the title. If I encode the address of people (the cities they live in) with a target encoder, do I still need to normalize that column? Of course, the capital is going to have more citizens ...
lte__'s user avatar
  • 1,310
1 vote
1 answer
82 views

How To Normalize Feature That Depends Arbitrarily On Date Published

Working on an ML project to predict the number of listens a certain podcast episode of my podcast will get in the first 28 days. The problem is that when I first started recording the podcast would ...
Adam Weitzman's user avatar
0 votes
1 answer
174 views

Normalize new data and old model

I'm playing with machine learning and LSTM. My goal is to learn something new and work with real data. Currently, I'm trying to predict bitcoin price. I have understood the necessity to normalize data,...
Oyabi's user avatar
  • 101
0 votes
0 answers
35 views

Impact of log transformation and Normalisation in the context of EDA and ML

Is data normalisation an alternative for log transformation? I understand that both helps us to normalisation helps me to make my distribution gaussian. Thanks in advance for your help!
kurmanjo's user avatar
0 votes
0 answers
2k views

Normalizing images with OpenCV (divide by 255)

I'm loading images from my dataset, which are all of resolution 200x200 and in RGB format. I'm loading them using OpenCV for Python, with the following code: ...
Tegig's user avatar
  • 1
0 votes
0 answers
394 views

Image normalization and reverse normalization: colors lost on image generation (GAN)

I'm working on a Gan. Based on different papers, I use a Tanh activation function on the last layer of the generator. Which produces [-1,1] outputs. To make this coherent, I use image normalization ...
Bouji's user avatar
  • 3

1
2 3 4 5 6