Questions tagged [normalization]

Normalization is a technique often applied as part of data preparation for machine learning. The goal of normalization is to change the values of numeric columns in the dataset to use a common scale, without distorting differences in the ranges of values or losing information.

Filter by
Sorted by
Tagged with
1
vote
0answers
27 views

When use standardization, normalization or both?

I have a dataset with variables with different scales as shown in the figure below. I need to group individuals together and I'm testing algorithms like Kmeans and DBScan. In all tests I'm extracting ...
1
vote
1answer
14 views

Correct approach to scale (min-max scaler) both input and output signal data for unsupervised learning?

I am working on a denoising autoencoder problem with noisy and clean signals. Before I pass the signals to my model I want to apply min-max normalization and am unsure of the correct way to apply this....
0
votes
1answer
21 views

How to deal with data having 0 values in many columns?

I am trying to implement logistic regression but the dataset that I have have many columns with skewed data and most of them have 0 as values. I also the skewness of data for many columns its going ...
1
vote
1answer
23 views

Is standardization/normalization a good way of reducing the impact of outliers when I'm training a machine learning model?

Recently, I have read some papers in which the authors state that they have performed standardization/normalization of the variables for reducing the impact of outliers in the machine learning models ...
0
votes
1answer
24 views

How to visualize data after performing OneHotEncoding and normalization?

I have a dataset and on that, I have performed OneHotEncoding and Standardization using standard scalar, Now that I have preprocessed data I have to visualize it, but on converting it to pandas ...
0
votes
0answers
14 views

Minmaxscaler in C

I want to use minmaxscaler in C. Is there any lite packages or libs for C? Or if any suggestion on how I can implement the minmaxscaler in C?
1
vote
2answers
36 views

OneVsRest Classification why do the probabilites sum to 1?

I am using OneVsRest Classifier in sklearn. So a multilabel model, 4 models for each class (i have 4 classes). When i called the predict_proba method i therefore get an array with 4 columns each one ...
0
votes
0answers
9 views

Creating a popularity index from multivariate data

I have some data from an ecommerce website with features like product_name, product_category product_link, product_id, free_delivery(1 or 0), price, discount, avg_rating, number of reviews, ...
0
votes
1answer
11 views

What does Keras image generators do with input images samplewise_std_normalization= True?

I have trained a a convolutional network samplewise_std_normalization=True. Now I want to check my model in real-time using Opencv. Therefore I would like to perform the same preprocessing on the ...
1
vote
1answer
27 views

Normalization with learning/test dataset in [0,1]

Say you split your data into two sets: training and test sets. You know that the inputs of your data are in [lower_bounds, upper_bounds]. Now, assume that you would like to do a min-max normalization ...
1
vote
0answers
31 views

Plotting cosine similarities in 3d space from word embeddings

I'm working on a project in which I want to estimate biases from a large corpus of newspaper articles using word2vec. Following this and this paper, biases are calculated by constructing dimension x ...
0
votes
1answer
22 views

how to choose between data normalization or standadization?

I have been studying about data scaling. Two common methods for it are the StandardScaler and MinMaxScaler. As I understood, StandardScaler expects the data to be normally distributed, but I have seem ...
0
votes
1answer
13 views

How do you normalise the train+validation sets together?

This question is somewhat related to: Is it correct to join training and validation set before inferring on test-set? As far as I understand, normalisation in general is done in the following way: ...
1
vote
0answers
15 views

What is num_groups in GroupNorm and how to choose it

I found that batch_norm can cause problems with small batch sizes and that GroupNorm is a good alternative. Now, GroupNorm requires two parameters, the num_group and the num_channels. How can I choose ...
1
vote
1answer
75 views

sklearn MinMaxScaler: Inverse does not equal original

I am using MinMaxScaler on a large dataset (2201887, 3) to normalize features. Inversed values does not match originals. I tested with the target column, first (a), I applied the scaler on 10 values, ...
1
vote
1answer
24 views

Normalization of production data

When training a model we split the dataset into training set and test set. In case a normalization/standardization is needed on any column then this process is done separately for training set and ...
0
votes
1answer
21 views

Should I normalise my data if future unseen data may have a different range?

I'm new to ML and researching data prep, more specifically feature normalisation. My question is whether it's a good idea to normalise data when its range may change over time? For example, if I'm ...
0
votes
1answer
17 views

Normalize data between 0 and 95 instead of between 0 and 100

I want to normalize the data between 0 and 95 instead of 0 and 100. I am using this formula to normalize between 0 and 100, please let me know how to edit it. ...
0
votes
1answer
20 views

How should a stateless data transformation be applied in regard to train/test split?

I want to apply spatial sign transformation to my data, but unlike other transformations this one is stateless. I am using sklearn and normallly i would first use ...
0
votes
1answer
95 views

K-means clustering with categorical data

I am doing a clustering analysis using K-means and I have around 6 categorical variables that I want to consider in the model. When I transform these variables as dummy variables (binary values 1 - 0) ...
0
votes
0answers
20 views

Is it better to normalize raw data to [-1, 1]

I am working on Radio signal classification. I have data generated synthetically and not normalized with the following features: ...
0
votes
1answer
55 views

What is the best practice to normalize/standardize imbalanced data for outlier detection or binary classification task?

I'm researching Anomaly/outlier/fraud detection, and I'm looking for the best practice to pre-process the synthetic data for imbalanced data. I have checked all methodology for normalizing/...
1
vote
0answers
33 views

Implementation of an Adaptive Normalization method

Referencing: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.718.9985&rep=rep1&type=pdf I'm trying to wrap my head around the method described in this paper I currently have 2 main ...
2
votes
1answer
166 views

Normalization vs standardization for image classification problem

For day and night image classification, is it better to normalize or standardize images? In general, when should I use each method? I am interested in with example why one method is preferred over ...
0
votes
0answers
13 views

How to Normalize image intensities of CT and MRI images (single channel)

I have data-set which contains MRI and CT images and all of them are labeled. I want to create MRI-CT classifier. But intensity range of MRI and CT are different. CT ranges between (-1024 and 2000) ...
0
votes
0answers
34 views

Input pipeline with an autoencoder and tf.data

I am using an autoencoder to detect anomalies in dataset of network traffic. The dataset is a csv file, and is loaded and preprocessed with pandas (encoded categorical features with pandas.get_dummies(...
0
votes
1answer
13 views

Normalization of possibly not fully representative data

I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:...
0
votes
1answer
46 views

Is it possible to do Normalization before Xgboost?

Currently I am working on a project which uses Xgboost Regression. Before putting data into model, I implemented Normalization, the accuracy significantly increased compared with without Normalization....
1
vote
1answer
19 views
1
vote
1answer
49 views

Why my regression model always be dominanted by one feature?

I am working on a financial predict problem. which means it is a time series prediction problem. I have three features, which have high correlation(each two's corr is about 0.6) And I do the linear ...
0
votes
0answers
23 views

Normalization during Inference in Keras

I am training a binary neural network so I can obtain its weights, and use them on another network where I add a few things. For example, this this one layer of the network: ...
0
votes
0answers
52 views

Normalization in Neural Networks Regression in MATLAB

I have 20 samples for a regression fitting purpose in Neural Network Toolbox (nnstart, nftool) in MATLAB . I have 3 inputs of 20 samples each (3 X 20 ) and 1 output (3 X 20, as the output number is ...
1
vote
0answers
168 views

Sliding Window Normalization

I've been reading about time series forecasting and many approaches use a sliding window method. What I don't get about it is how to properly normalize your data. Most codes I've seen so far normalize ...
0
votes
1answer
28 views

Normalized 2D tensor values are not in range 0-1

Below function takes in 2D tensor and normalizes it using broadcasting .The issue is except all values to be in range 0-1 but the result has values outside this range . How to get all values in 2D ...
1
vote
1answer
72 views

What is the difference between normalization and re-scaling?

This site does not describe the nature of the normalization tag. How does it differ from re-scaling? Many authors use the two terms interchangeably. I can not understand normalization's operational ...
1
vote
0answers
33 views

Considerations to take into account when clustering

My idea is to use clustering to perform stock segmentation based on risk, building different risk levels that might adapt better to different kind of users. Hence I have computed different risk ...
1
vote
1answer
70 views

Why does my manual derivative of Layer Normalization imply no gradient flow?

I recently tried computing the derivative of the layer norm function (https://arxiv.org/abs/1607.06450), an essential component of transformers, but the result suggests that no gradient flows through ...
0
votes
0answers
41 views

Why does normalization improve my decision tree performances?

I have a regression problem for which I have to try several models, so I normalized my data and then tried to use a decision tree regressor (from sklearn.tree) and I noticed very good results (...
0
votes
1answer
87 views

Normalization for a 2d input array

I am new to machine learning and trying to apply it to my problem. I have a training dataset with 44000 rows of features with shape 6, 25. I want to build a sequential model. I was wondering if there ...
3
votes
2answers
231 views

Layer normalization details in GPT-2

I've read that GPT-2 and other transformers use layer normalization before the self-attention and feedforward blocks, but I am still unsure exactly how the normalization works. Let's say that our ...
1
vote
0answers
27 views

Normalizing variables with logarithmic shape

A simple model with two variables [A,B] to train, let's say, a logistic regression or any other classification model: A: Flat distribution from 0 to 100. B: A logarithmic distribution from 0 to a few ...
1
vote
0answers
51 views

whether to use normalization while performing PCA?

I have an excel file containing a table where I registered the frequency of three linguistic phenomena in 72 poems. Since the poems have different lengths I normalized the results dividing each value ...
0
votes
0answers
7 views

Encoding Data and huge loss during ANN training

I just started to learn on ANN and tried to experiment on my own on a Linear Regression. I got a dataset which had housing prices for a city. Tried going through this but my model gives me a huge loss....
0
votes
0answers
14 views

estimate user's satisfaction of a video based on how much of it they watched - normalization

I am trying to estimate how much a user liked a video using how much of the video they watched. Let's say, on the scale of 1 to 10, 1 means that the user didn't like it at all, and 10 means they ...
0
votes
0answers
4 views

While Merging image datasets which of the image parameters should be prepossessed/Normalized before giving to a CNN Model?

When two datasets are merged or images of different parameters size, dimension, Format are combine which parameters of the datasets should be normalized/ pre-processed before giving it to a model?
0
votes
1answer
23 views

Cleaning NaNs with averages pre or post split? [duplicate]

I have a column with some NaNs in it and I want to replace those NaNs with the average/median/mode. Technically, the validation/ test data has never been seen before - so how could I include it in the ...
0
votes
0answers
80 views

Is normalizing term weight necessary when cosine similarity is used in retrieval?

When using cosine similarity in information retrieval, document vector length and query vector length are used for normalization. So if TF-IDF is used as a weighting function, then using raw frequency ...
0
votes
0answers
31 views

Scalling features for competition participants

Hello there and Happy Holidays. I have a data set with each row representing a race with 6 participants, with each participant having its own column for each feature. The target variable is ...
0
votes
0answers
26 views

Normalization of encoded feature?

I am a beginner in ML, and I am working on a classification problem on big data (its shape is (8921483, 52)) which its features are mostly categorical. One of the features has 175365 different ...
1
vote
1answer
193 views

First perform data augmentation or normalization?

Should I first perform data augmentation or normalization in deep learning? I am mainly interested in 2D and 3D input data. In tutorials that I have seen so far the data augmentation always comes ...

1
2 3 4 5