Skip to main content

Questions tagged [normalization]

Normalization is a technique often applied as part of data preparation for machine learning. The goal of normalization is to change the values of numeric columns in the dataset to use a common scale, without distorting differences in the ranges of values or losing information.

Filter by
Sorted by
Tagged with
0 votes
0 answers
22 views

Negative values when calculating weighted Jaccard similarity

I have a bilateral dataset that includes countries and their the weight of their relation. I'd like to calculate the similarity of countries in 1) who their trade parterns are and 2) the weight of the ...
Johanna's user avatar
0 votes
0 answers
5 views

Can principal components changed by a normaliaation method be used to construct original data shape with SVD

I would like to use an algorithm called Harmony to normalize my data. Harmony takes as input principal components ($PC$), and ...
MadmanLee's user avatar
  • 101
0 votes
1 answer
72 views

Min-Max Scaling more sensitive to outliers than 'Simple Feature Scaling'?

I am confused as to the pros and cons of two different approaches to normalization: Min-Max Scaling, and what the lecturer in the course I am taking refers to as 'Simple Feature Scaling'. The latter ...
Chris Bedford's user avatar
0 votes
2 answers
80 views

batch Normalization and Layer Normalization difference

In Batch Normalization, mean and standard deviation are calculated feature wise and normalization step is done instance wise and in Layer Normalization mean and standard deviation are calculated ...
April's user avatar
  • 1
1 vote
1 answer
56 views

How to normalize the features without the knowledge of the min and max values in online learning?

I am developing an online learning platform where input features are gathered from various sensors. However, these features may have vastly different ranges. For example, displacement values may be ...
JJbow's user avatar
  • 15
0 votes
0 answers
21 views

Should I log transform the data if the dataset is small?

Suppose I have a dataset with only about 25 rows. All the relevant columns are all integers. The ranges of them go from 1 million to 100 million, and the distributions are all skewed right. In this ...
DSGuest's user avatar
1 vote
0 answers
32 views

Standardizing my target versus not-standardizing

I've heard from multiple sources that it depends on whether I should standardize or not. Most of the time, people would say it doesn't make sense to do so, some would say it's better if I standardize ...
Justin Jonany's user avatar
0 votes
0 answers
25 views

TFRobertaSequenceClassification for Address Normalization task

I have dataset with two column: one with faulty addresses, and other with correct addresses. I want to train a model such that, I can use it later for correcting all the incoming faulty addresses. I ...
learner_account's user avatar
0 votes
0 answers
24 views

Data Preprocessing in the Wild?

I am new to ML, NN, and data science as a whole so the following question might sound silly. How can we perform inference when the model is deployed in the wild? To my understanding, cleaning/...
Abdullah Abdulrahman's user avatar
1 vote
2 answers
107 views

weighting voting classifier (MAE and MSE)

I am trying to optimize the weights of a Voting Regressor problem. To achieve the best score, I am considering both MAE and MSE as parameters, using the following formula: score = w * MAE + (w-1) * ...
Guilherme Raibolt's user avatar
0 votes
0 answers
14 views

Comparing multiple multivariate datasets

Take the two datasets below: default rate state age income asofdate 10 Texas 55 100,000 202309 14 Texas 35 97,000 202309 18 Texas 55 95,000 202308 22 Texas 35 95,000 202308 8 New York 21 55,000 ...
rpiston's user avatar
0 votes
0 answers
10 views

How to account for exponential oversampling?

I have a dataset of frequencies from 20Hz to 20kHz. The measurement rig that created the dataset doesn't sample evenly across this frequency range. There is a much higher density of sampling happening ...
db2791's user avatar
  • 101
0 votes
0 answers
18 views

Riemannian metric in Layer Normalization

I'm reading a paper about Layer normalization, and I couldn't find any clear explanation for this part: Q1. Can anyone describe the derivation of the first equation in (8)? Q2. I cannot understand ...
user154010's user avatar
0 votes
1 answer
216 views

Histogram of Oriented Gradients (HOG) - Why normalize 16x16 blocks and not the whole picture?

I'm trying to learn Histogram of Oriented Gradients (HOG) I understand why we compute the gradient and the orientation and also map every gradient into a 9 binaries histogram that spans from 0 to 180. ...
euraad's user avatar
  • 115
0 votes
2 answers
631 views

What information is lost due to normalization?

Normalization is usually seen as a good thing for data preprocessing before training a model. But I am wondering if there are some information in the data that might be lost during this process. The ...
Fraïssé's user avatar
  • 119
0 votes
1 answer
272 views

How would normalizing be affected by outliers? And how to avoid it?

I have a data set that boils down to Three clomuns: 1.Supplier name 2. Number of transactions with supplier 3. Total value of those transaction. I'm trying to find the best way to rank all suppliers ...
Rakuzan's user avatar
0 votes
1 answer
83 views

Converting categorical to the percentage

How do I convert the categorical value to the percentage?| I have this asset wellness data: Poor: 3 Warning: 27 Good: 120 How do I convert it to the percentage ...
Muhammad Ikhwan Perwira's user avatar
1 vote
1 answer
72 views

Should I normalize entire dataset or normalize every x_train and every y_train for timeseries forecasting LSTM?

Suppose I have list like this: ...
Muhammad Ikhwan Perwira's user avatar
2 votes
2 answers
707 views

Why are normal distributions so important in deep learning?

I am currently reading on normalization/standardization techniques as well as batch normalization in deep learning and I don't really understand why normal distributions are so important inside deep ...
Kiran Manicka's user avatar
0 votes
1 answer
434 views

Why apply min-max normalization to each individual mel spectrogram for a training set?

I am watching a tutorial on using mel spectrograms to classify the audio's genre via CNN. My question is why apply local min-max normalization to each individual mel spectrogram? What I mean by local ...
Hayden LaBrie's user avatar
1 vote
0 answers
17 views

How to normalize(or other) the audio data so that the same labels with the similar characteristics from different records?

I am trying to detect swallows from recordings taken from hospital. I manually labelled the recordings on the Praat. Now the valid labels are silence, swallows and nonswallows(noise, enviromenment ...
Yalçın Cenik's user avatar
0 votes
0 answers
86 views

Classical Perceptron Algorithm taking too long to evaluate on un-normalized data

I need to implement classical perceptron algorithm from scratch using numpy and pandas for an assignments. I have done so using this algorithm: I have a linearly seperable dataset of 568 rows and 30 ...
Prad's user avatar
  • 1
1 vote
0 answers
101 views

NANs, Infinities, and very large losses with normalizing flows

I am new to normalizing flows and have been trying to use them with a high-dimensional dataset, and I have been running into very large numbers and errors with sampling that don't occur when I use a ...
quail's user avatar
  • 31
0 votes
2 answers
29 views

How will a model handle real-life values in real-life applications without scaling?

I am learning ML and facing confusion about data scaling. For example, I have the following data: Weight(KG) Balance($) 75 3401542 99 4214514 Now, if I use StandardScaler, I may get something like ...
Ishrat Hossain's user avatar
0 votes
1 answer
66 views

Normalization / Overfitting Issues

I have a dataset with 608 inputs and I'm trying to output a single 1 or 0 result. My validation data has 69.12% 0's. When ran, my model always returns 69.12% accuracy, presumably because it's "...
n3uralio's user avatar
1 vote
1 answer
69 views

Dictionary-based text analysis- dealing with length

I am working on an analysis using a dictionary-based text-as-data approach. I have a dataset of texts (n=1200), and I am applying a dictionary of 50 words (I tokenize the text with each word being one ...
Iamembarassed123's user avatar
0 votes
1 answer
49 views

Python dataset normalization for convolutional autoencoder

I have a csv files which contain pixel neighboorhood information. Here an example of the dataset: ...
user979974's user avatar
0 votes
1 answer
106 views

Normalising data for simple linear regression

Consider a simple linear regression problem where: X = [1,2,3,4,5,100,200] Y= [2,4,6,8,10,200,400] Clearly, the relationship is of the form $y=2x$; While trying ...
butwhy's user avatar
  • 33
3 votes
1 answer
1k views

Out of range [0,1] MinMaxScaler for test data

I know that for MinMaxScaler we should apply it to train data, then apply it, with the obtained parameters, over test data: ...
John mx's user avatar
  • 31
0 votes
0 answers
186 views

How to use MinMaxScaler when X_train and X_test are different sizes

I'm using an LSTM through Keras to predict a time series. My inputs are the previous measurements of one time series(v1) at time 0-59 seconds and the goal is to predict the measurement of a different ...
user142789's user avatar
0 votes
0 answers
28 views

What is the NLP task that convert "your" to "one's", "is" to "be"?

In lemmatization, "is" remains the same; in stemming, 'your' remains the same; text normalization is irrelevant. It can be solved using rules(mapping, like "my" -> "one's&...
Lerner Zhang's user avatar
0 votes
1 answer
75 views

Normalising Image Data

Hi I am wondering when it comes to normalising images across each of the channels, do you use the same scaling factors that is used for training for testing set as well or separate ones. In ...
imantha's user avatar
  • 103
0 votes
1 answer
530 views

Standardization after log transformation

I have a few question about log transformation and standardization. First: Should I standardize my features after doing log transformation? Second: I still do not understand, because when doing log ...
Jovian Aditya's user avatar
1 vote
1 answer
68 views

How to do an incremental update for the mean and standard deviation of tensor data?

I have a big dataset (some 400Gb) consisting of tensor data (shape is $(600, 600, 10)$) and I want to normalize this dataset before feeding it to a neural network but this dataset can't fit in my ...
Toshi Mint's user avatar
0 votes
2 answers
40 views

Normalization of predictors

Does normalization (ex: log or sqrt) of skewed features help to reduce model's bias? or it is better to leave the skewed distribution of predictors as it is?
Kusisi Karem's user avatar
0 votes
1 answer
37 views

How "normal" should my input data be?

When training a neural net, I understand the value in normalising the input data to have mean = 0 and stdev = 1 (standardising the data). But I often see people make the data even more "normal&...
JessicaDavies's user avatar
3 votes
1 answer
123 views

What justifies feature scaling?

Although I can understand the significance of feature scaling in some cases (e.g. when gradient descent is involved), I don't feel I understand the necessity of this process in general. But there a ...
ado sar's user avatar
  • 191
0 votes
1 answer
91 views

why we have problem with gradients when feature values are of different range?

A blog below mentioned. " Because different features do not have similar ranges of values, gradients may take a long time, oscillate back and forth, and take a long time before they can finally ...
smsubham's user avatar
0 votes
2 answers
142 views

How to convert city name prefix abbreviations?

Is there any standard tool or library or list for expanding town name abbreviations? For example "MT HOLLY" -> "MOUNT HOLLY" or "ST MICHAELS" -> "SAINT ...
Cerin's user avatar
  • 121
2 votes
0 answers
48 views

When is scaling and centering important?

There are some models such as PCA or SVM where scaling and centering of training data is essential. There are some models, mostly tree-based where scaling and centering is not required at all. I don't ...
xan's user avatar
  • 121
0 votes
0 answers
158 views

Why input normalization leads to worse performance?

I'm building quite simple NN with Dense layers followed by ReLU activations and I noticed something unexpected. Generally, I've been confident that normalizing the input to have mean of 0 and standard ...
Xaume's user avatar
  • 192
0 votes
0 answers
168 views

Do I need to encode numerical variables like "year"?

I have a simple time-series dataset. it has a date-time feature column. ...
smarks70's user avatar
0 votes
2 answers
178 views

Standardization in combination with scaling

Would it be ok to standardize all the features that exhibit normal distribution (with StandardScaler) and then re-scale all the features in the range 0-1 (with <...
Caterina's user avatar
  • 119
0 votes
1 answer
32 views

Normalizing data from same variable but different individuals

I'm new to machine learning. I have the following scenario: I have five individuals that are each carrying an accelerometer. That sensor measures movement/acceleration on a scale from 0 to 255, 0 ...
Benisburgers's user avatar
1 vote
0 answers
56 views

Normalization and Denormalization

I have few queries. 1) Is normalization required for ANN / CNN /LSTM ? 2) If we normalize the data with MinMax Scaler, then in that case how to denormalize it and when to denormalize it so that we ...
Tarun Sharma's user avatar
0 votes
0 answers
238 views

How to save pixels after normalization

I want to normalize my images and use them in the training. But I couldn't find a way to save images after making changes below...How can I save it? ...
Zehra N.'s user avatar
0 votes
2 answers
312 views

Is normalization needed for TargetEncoded Variables?

Basically the title. If I encode the address of people (the cities they live in) with a target encoder, do I still need to normalize that column? Of course, the capital is going to have more citizens ...
lte__'s user avatar
  • 1,340
1 vote
1 answer
93 views

How To Normalize Feature That Depends Arbitrarily On Date Published

Working on an ML project to predict the number of listens a certain podcast episode of my podcast will get in the first 28 days. The problem is that when I first started recording the podcast would ...
Adam Weitzman's user avatar
0 votes
1 answer
198 views

Normalize new data and old model

I'm playing with machine learning and LSTM. My goal is to learn something new and work with real data. Currently, I'm trying to predict bitcoin price. I have understood the necessity to normalize data,...
Oyabi's user avatar
  • 101
0 votes
0 answers
36 views

Impact of log transformation and Normalisation in the context of EDA and ML

Is data normalisation an alternative for log transformation? I understand that both helps us to normalisation helps me to make my distribution gaussian. Thanks in advance for your help!
kurmanjo's user avatar

1
2 3 4 5 6