Questions tagged [normalization]

Normalization is a technique often applied as part of data preparation for machine learning. The goal of normalization is to change the values of numeric columns in the dataset to use a common scale, without distorting differences in the ranges of values or losing information.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Proper iteration over time series data for LSTM neural network

I’m using the supervised learning method with an LSTM network to predict forex prices. To achieve this I’m using deeplearning4j library but I doubt several points of my implementation. I turned off ...
user avatar
0 votes
0 answers
31 views

Do I need to encode numerical variables like "year"?

I have a simple time-series dataset. it has a date-time feature column. ...
user avatar
0 votes
0 answers
10 views

Would I be able to combine features on a different unit scale after normalizing?

I'd like to explore some interactions between my variables but they're on different measurement scales. Would for example the absolute value of the difference of them after scaling make sense? From ...
user avatar
  • 3
0 votes
2 answers
36 views

Standardization in combination with scaling

Would it be ok to standardize all the features that exhibit normal distribution (with StandardScaler) and then re-scale all the features in the range 0-1 (with <...
user avatar
  • 109
0 votes
1 answer
17 views

Normalizing data from same variable but different individuals

I'm new to machine learning. I have the following scenario: I have five individuals that are each carrying an accelerometer. That sensor measures movement/acceleration on a scale from 0 to 255, 0 ...
user avatar
1 vote
0 answers
8 views

Normalization and Denormalization

I have few queries. 1) Is normalization required for ANN / CNN /LSTM ? 2) If we normalize the data with MinMax Scaler, then in that case how to denormalize it and when to denormalize it so that we ...
user avatar
0 votes
0 answers
14 views

How to save pixels after normalization

I want to normalize my images and use them in the training. But I couldn't find a way to save images after making changes below...How can I save it? ...
user avatar
0 votes
1 answer
27 views

Is normalization needed for TargetEncoded Variables?

Basically the title. If I encode the address of people (the cities they live in) with a target encoder, do I still need to normalize that column? Of course, the capital is going to have more citizens ...
user avatar
  • 1,248
0 votes
0 answers
14 views

Clarify normalization methods

I'm following this guide on detecting anomalies using autoencoders. The section titled "Normalising & Standardising" seems to be describing normalization in terms of scaling and shifting ...
user avatar
  • 111
0 votes
0 answers
7 views

Handling bias inputs during normalization

Suppose I have an input matrix $\mathbf X\in \mathbb R^{(D+1)\times N}$ where $N$ is number of samples $D$ is dimension of an input vector $x$ and extra $1$ dimension is for bias where all bias ...
user avatar
1 vote
1 answer
12 views

How To Normalize Feature That Depends Arbitrarily On Date Published

Working on an ML project to predict the number of listens a certain podcast episode of my podcast will get in the first 28 days. The problem is that when I first started recording the podcast would ...
user avatar
0 votes
1 answer
8 views

Is It Okay To Do 0-1 Scaling Then Divide By The Standard Deviation?

If am understanding stuff correctly, if I have a df I can first do 0-1 scaling on it to get equal ranges while preserving the data series's original means and standard deviations and then once I ...
user avatar
0 votes
1 answer
83 views

Normalize new data and old model

I'm playing with machine learning and LSTM. My goal is to learn something new and work with real data. Currently, I'm trying to predict bitcoin price. I have understood the necessity to normalize data,...
user avatar
  • 101
0 votes
0 answers
19 views

Image-wise vs. pixel-wise of the CT images for segmentation task

I'm working on a semantic segmentation of a lung region on a computer tomography (CT) images. CT images have only 1 channel (Hounsfield units) and can be put in one class of images (one distribution) ...
user avatar
0 votes
0 answers
10 views

What kind of impact has a standardization on our ML model?

I understand that standardisation helps to compare two different normal distributions (e.g. performance of students in cambridge vs. stanford) and it also helps find probabilites by using the z score ...
user avatar
0 votes
0 answers
17 views

Impact of log transformation and Normalisation in the context of EDA and ML

Is data normalisation an alternative for log transformation? I understand that both helps us to normalisation helps me to make my distribution gaussian. Thanks in advance for your help!
user avatar
0 votes
0 answers
265 views

Normalizing images with OpenCV (divide by 255)

I'm loading images from my dataset, which are all of resolution 200x200 and in RGB format. I'm loading them using OpenCV for Python, with the following code: ...
user avatar
  • 1
0 votes
0 answers
61 views

Image normalization and reverse normalization: colors lost on image generation (GAN)

I'm working on a Gan. Based on different papers, I use a Tanh activation function on the last layer of the generator. Which produces [-1,1] outputs. To make this coherent, I use image normalization ...
user avatar
  • 3
0 votes
0 answers
39 views

Detection of unloadings by GPS coordinates

I have a history of the car's movements, a list of GPS coordinates with timestamp (in GPX format). I'm new to ML, tried to solve but doesn't work well. I have several problems: How to correctly ...
user avatar
  • 101
-1 votes
1 answer
23 views

Normalizing Distributions for features in predicting

Should features used for predictions be normalized if they are highly skewed. Or should I only normalize target feature that is supposed to be predicted?
user avatar
  • 33
1 vote
1 answer
62 views

T-test against normalised or standardised data gives different results

I am studying the problem to predict popularity of a tweet, and want to test null hypothesis: there is no relationships between favorite_counts and another set of variables, like number of friends of ...
user avatar
1 vote
0 answers
17 views

How to convert varians of a character to the same character in python? [closed]

How to convert variants of a character to a same character in python. For example, the character a with the U+0061 Unicode has ...
user avatar
1 vote
1 answer
30 views

Why feature normalization can increase the biometric recognition accuracy?

In a biometric recognition system, I have noticed that normalizing the extracted wavelet features leads to increasing the recognition accuracy. The classifier used is K-nearest neighbor (KNN), and ...
user avatar
  • 111
3 votes
1 answer
56 views

How do I normalize json data into pandas (Covid-19 data) [closed]

I am trying to import all up-to-date datasets in JSON format on the covid-19 pandemic into a pandas dataframe. I believe it should be possible by using ...
user avatar
0 votes
0 answers
13 views

Should I normalise or detrend time series data before creating MLP models

Am building MLP models on forecasting timeseries data. Am new in the field of machine learning and I have read about Detrending and normalisation. So which method (normalisation or detrending) will be ...
user avatar
0 votes
1 answer
49 views

Two-parametric transformation of Box-Cox vs Yeo–Johnson transformation

I choose which transformation to use for my data (data contains both positive and negative values). Wikipedia says the following: The Yeo – Johnson transformation allows also for zero and negative ...
user avatar
  • 89
1 vote
0 answers
57 views

How to combine data from multiple Google Trends queries effectively?

As you might know, Google Trends works by normalising a random sample of the search term data, with the sample changing at least once per day, from my experience. This is not an issue for western ...
user avatar
0 votes
1 answer
26 views

Applying standardization using ImageDataGenerator

I have a multiclass image dataset ( 8 classes) that is divided as follows, the main folder is called training and I have 8 subfolders with each subfolder for one class. I know how to perform data ...
user avatar
  • 7
0 votes
0 answers
25 views

Should non-stationary time series be differenced when fit through neural networks?

I am fitting a recurrent neural network (RNN) on some non-stationary time series data. I know that, in the case of linear models, it is common practice to difference the series in order to make them ...
user avatar
  • 11
0 votes
1 answer
47 views

How use logistic function to normalize data to (0,1)

I am reading paper about data normalization and I am interested how is it possible to use the logistic sigmoid function to normalize data to the specific interval (0,1). There is only short mention in ...
user avatar
  • 103
1 vote
0 answers
51 views

When use standardization, normalization or both?

I have a dataset with variables with different scales as shown in the figure below. I need to group individuals together and I'm testing algorithms like Kmeans and DBScan. In all tests I'm extracting ...
user avatar
  • 21
1 vote
1 answer
40 views

Correct approach to scale (min-max scaler) both input and output signal data for unsupervised learning?

I am working on a denoising autoencoder problem with noisy and clean signals. Before I pass the signals to my model I want to apply min-max normalization and am unsure of the correct way to apply this....
user avatar
  • 13
0 votes
1 answer
43 views

How to deal with data having 0 values in many columns?

I am trying to implement logistic regression but the dataset that I have have many columns with skewed data and most of them have 0 as values. I also the skewness of data for many columns its going ...
user avatar
1 vote
1 answer
289 views

Is standardization/normalization a good way of reducing the impact of outliers when I'm training a machine learning model?

Recently, I have read some papers in which the authors state that they have performed standardization/normalization of the variables for reducing the impact of outliers in the machine learning models ...
user avatar
  • 127
0 votes
1 answer
37 views

How to visualize data after performing OneHotEncoding and normalization?

I have a dataset and on that, I have performed OneHotEncoding and Standardization using standard scalar, Now that I have preprocessed data I have to visualize it, but on converting it to pandas ...
user avatar
0 votes
0 answers
19 views

Minmaxscaler in C

I want to use minmaxscaler in C. Is there any lite packages or libs for C? Or if any suggestion on how I can implement the minmaxscaler in C?
user avatar
  • 1
1 vote
2 answers
87 views

OneVsRest Classification why do the probabilites sum to 1?

I am using OneVsRest Classifier in sklearn. So a multilabel model, 4 models for each class (i have 4 classes). When i called the predict_proba method i therefore get an array with 4 columns each one ...
user avatar
  • 466
0 votes
0 answers
12 views

Creating a popularity index from multivariate data

I have some data from an ecommerce website with features like product_name, product_category product_link, product_id, free_delivery(1 or 0), price, discount, avg_rating, number of reviews, ...
user avatar
  • 101
0 votes
1 answer
100 views

What does Keras image generators do with input images samplewise_std_normalization= True?

I have trained a a convolutional network samplewise_std_normalization=True. Now I want to check my model in real-time using Opencv. Therefore I would like to perform the same preprocessing on the ...
user avatar
1 vote
1 answer
41 views

Normalization with learning/test dataset in [0,1]

Say you split your data into two sets: training and test sets. You know that the inputs of your data are in [lower_bounds, upper_bounds]. Now, assume that you would like to do a min-max normalization ...
user avatar
  • 13
1 vote
0 answers
49 views

Plotting cosine similarities in 3d space from word embeddings

I'm working on a project in which I want to estimate biases from a large corpus of newspaper articles using word2vec. Following this and this paper, biases are calculated by constructing dimension x ...
user avatar
0 votes
1 answer
35 views

how to choose between data normalization or standadization?

I have been studying about data scaling. Two common methods for it are the StandardScaler and MinMaxScaler. As I understood, StandardScaler expects the data to be normally distributed, but I have seem ...
user avatar
0 votes
1 answer
24 views

How do you normalise the train+validation sets together?

This question is somewhat related to: Is it correct to join training and validation set before inferring on test-set? As far as I understand, normalisation in general is done in the following way: ...
user avatar
  • 3
0 votes
0 answers
34 views

What is num_groups in GroupNorm and how to choose it

I found that batch_norm can cause problems with small batch sizes and that GroupNorm is a good alternative. Now, GroupNorm requires two parameters, the num_group and the num_channels. How can I choose ...
user avatar
0 votes
1 answer
512 views

sklearn MinMaxScaler: Inverse does not equal original

I am using MinMaxScaler on a large dataset (2201887, 3) to normalize features. Inversed values does not match originals. I tested with the target column, first (a), I applied the scaler on 10 values, ...
user avatar
1 vote
1 answer
59 views

Normalization of production data

When training a model we split the dataset into training set and test set. In case a normalization/standardization is needed on any column then this process is done separately for training set and ...
user avatar
  • 11
0 votes
1 answer
32 views

Should I normalise my data if future unseen data may have a different range?

I'm new to ML and researching data prep, more specifically feature normalisation. My question is whether it's a good idea to normalise data when its range may change over time? For example, if I'm ...
user avatar
1 vote
1 answer
183 views

Normalize data between 0 and 95 instead of between 0 and 100

I want to normalize the data between 0 and 95 instead of 0 and 100. I am using this formula to normalize between 0 and 100, please let me know how to edit it. ...
user avatar
0 votes
1 answer
34 views

How should a stateless data transformation be applied in regard to train/test split?

I want to apply spatial sign transformation to my data, but unlike other transformations this one is stateless. I am using sklearn and normallly i would first use ...
user avatar
  • 115
0 votes
1 answer
339 views

K-means clustering with categorical data

I am doing a clustering analysis using K-means and I have around 6 categorical variables that I want to consider in the model. When I transform these variables as dummy variables (binary values 1 - 0) ...
user avatar

1
2 3 4 5 6