Questions tagged [feature-scaling]

Feature scaling is a data pre-processing step where the range of variable values is standardized. Standardization of datasets is a common requirement for many machine learning algorithms. Popular feature scaling types include scaling the data to have zero mean and unit variance, and scaling the data between a given minimum and maximum value.

Filter by
Sorted by
Tagged with
2
votes
1answer
26 views

Overfitting with sklearn pipeline - reasons why?

So.... I've been playing around with this for FAR TOOO LONG now and I really need some advice. Most people on kaggle concat training and testing set TOGETHER and then pre scale the data, this seems ...
0
votes
0answers
11 views

transformation and standardization. what will be the order?

I have a dataset with all positive numeric values for a classification problem. out of 8 columns 4 columns have skewed distribution. What will be the ideal order to follow ? ...
0
votes
0answers
24 views

Normalizing Feature/Label with Negative Values

I am creating a neural network using tensorflow that predicts the energy consumption of a vehicle. Originally, I planned on normalizing all of the features from 0 to 1 using the scikit-learn object ...
1
vote
1answer
16 views

Standardization of variables in log-scale

I have a doubt regarding standardization, I have to use a multivariate regression and one of the variables is in log-scale. Is using standardization enough to re-scale the log variable, or is it ...
1
vote
3answers
89 views

How to handle features with very broad range

I have a long list of continuous values like in the image below: The plot looks like this: How to handle such features? If I train the model with this, the model will not have the best precision, ...
1
vote
3answers
55 views

why to use Scaler.fit only on x_train and not on x_test for normalizing value using MinMaxScaler?

while normalising the data everone is saying that we need to fit only on x_train and not on x_test ? why is that we should not fit x_test ? if we should not fit the scaler on x_test then why we need ...
4
votes
2answers
68 views

Why Decision Tree Classifier is not working with categorical value?

I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx. I am trying to model a Decision Tree Classifier as part of an ...
0
votes
1answer
17 views

Does Sklean's SGDClassifier automatically standardize the training data when regularization is turned on?

Generally speaking--it is best to apply standarizaton (z-scoring the training data) prior to regularization. Does sklearn.linear_model.SGDClassifier automatically standardize the training data or not ...
1
vote
1answer
31 views

One scaler for all features or one scaler per feature?

I have a time series with more than 30 features. For preprocessing with scikit learn do you usually use one scaler per feature or one scaler for all features that should be standardized/normalized?
1
vote
2answers
65 views

why does making the target variable normally distributed helps?

while working on some regression problems I have found that if the target variable is skewed, making it normally distributed(using transformations) almost always helps. Why is that? Should we also ...
9
votes
6answers
221 views

When should I NOT scale features

Feature scaling can be crucially necessary when using distance-, variance- or gradient-based methods (KNN, PCA, neural networks...), because depending on the case, it can improve the quality of ...
0
votes
1answer
54 views

Feature transformation possible at selected features or only at all?

I want to cluster. I have different features for that. Some features have a very small value range (from 0 to 0.8) and some have a very large value range (from 0 to 5 million). I want to use the ...
2
votes
2answers
64 views

Does Feature Normalization affect Gradient Descent | Linear Regression

am new to datascience and i want to learn linear regression so i coded linear regression from scratch and performed gradient descent to find the best $w_\theta$ and $b_\theta$ values using a tutorial. ...
0
votes
1answer
36 views

How to scale a variable when not knowing the maximum

I have a dataset with different features where some of them are not categorical, so they need to be scaled or normalized (especially the target). However, normalizing between 0-1 for instance means ...
0
votes
2answers
87 views

how to use standardization / standardscaler() for train and test?

At the moment I perform the following: ...
2
votes
1answer
36 views

Standardscaler() not standardscaling?

I have following pipeline: estimators = [] estimators.append(('standardize', StandardScaler())) prepare_data = Pipeline(estimators) Originally, the data looks ...
0
votes
0answers
11 views

Standardizing in each fold - Learning Curve

Problem Description Hello, I have a classification problem and I want to perform cross validation (with hyper parameter tuning) in order to evaluate the generalization of my models. Basically the ...
2
votes
1answer
44 views

When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?

I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
0
votes
0answers
8 views

Applying different feature scaling to different features in the same dataset

I am currently studying a course about big data and data mining, so I am new to this field. We were tasked to make a model using a training dataset of 60 columns and 68 rows, and then use the test ...
0
votes
1answer
89 views

Correcting for one of multiple strong batch effects in a dataset

I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it ...
0
votes
0answers
19 views

Do not scale Hog features?

when I train LinearSVC with the Hog features extracted from the Fashion-MNIST dataset then I get better results if I don't use StandardScaler before training than I use it. ...
1
vote
0answers
14 views

Variable importance of Numerical features in Classification Model - Random Forest Classifier

I have few numeric features in my model. Out of 25 features, I have 7-8 numeric features in my model. One thing I observed is model gives more weightage to numerical feature compare to categorical ...
2
votes
2answers
44 views

Categorical features preprocessing for clustering

Can anyone tell suggest the best practice for clustering data with mixtured features (both with categorical and continuous). I am struggling with a problem; I realized that for all metrics algorithms ...
1
vote
1answer
85 views

Should scaling be done for mixed data (categorical and numerical)?

My dataset contains 13 attributes consisting of 10 Numerical and 3 Categorical attributes and Target. It has 180 observations ...
1
vote
0answers
13 views

can i scale features within each input catgory differently

i am trying to predict laptop prices from previous sales. based on a text description of the laptop condition and the sale price. the affect of the condition on the final price is going to be ...
1
vote
1answer
32 views

very important features, but rarely input

The features are of capital gain and capital loss, but very small amounts of people have one or the other. As far as I can see it may not signify any great difference, though I'm not sure how I would ...
0
votes
0answers
8 views

Scale a column with respect to the deviation in another

I have a dataset consisting of 644 features and the temperature that the features where captured at. I know that the temperature will effect the value of some of the features. Is it possible to scale ...
0
votes
2answers
267 views

Should we denormalize our data after normalization?

If we use sklearn library's preprocessing.normalie() function to normalize our data before learning, like this: ...
0
votes
0answers
45 views

What is the difference between row-wise and column-wise Z-score normalization?

I have a data set, each row represents a movie name, each column is a feature (such as genres), I want to perform cosine similarity to find out the similarity between each movie, before that I need to ...
2
votes
2answers
44 views

Training vs test data set for supervised learning in real life scenario

In the tutorials, I have noticed only similar data has been used with models training and prediction. I was wondering how cases where you can't find training data that is similar to your final use ...
1
vote
0answers
27 views

Python is reading my data with NANS and Infs, but they don't have any

I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
0
votes
1answer
337 views

Clustering with geolocation (lat/long pairs) attributes

I am trying to cluster customer behavior based on where they shop given by lat/long pairs. I also have other numeric attributes such as volume, average amount spent, etc. I am considering using ...
0
votes
0answers
21 views

What happens if you add a constant value to all input data points to neural networks?

I have a somewhat basic question about neural networks. What would be the effect on the performance of a neural network if we add a constant value to all data points? For example, suppose you have ...
1
vote
1answer
60 views

How to transform stock data for LSTM-based neural network

I am trying to classify stock returns using an LSTM-based neural network. I would like to use closing price and volume as features (see below), but am unsure of whether I need to transform these (e.g....
0
votes
1answer
53 views

Effect of skewness in data

I am preparing classification model. Many of numeric variables are positives skewed. Should I change a distribution of variables to be more Gaussian?
1
vote
1answer
592 views

Feature engineering for categorical variables

I have some categorical variables in my dataset for a regression problem. 1) One of the variable can take 3 values (Girls, Boys, Girls&Boys). Converting it into one-hot encoding or binary ...
1
vote
0answers
21 views

Transformation of non-categorical discrete feature

Goal: Predict a performance score of a place of interest in a given city based on (amongst others), the number of restaurants within 200m. $\\$ Dataset: $D$ with a feature $x$ indicating the $\...
0
votes
1answer
107 views

How to scale exponential data for a regression problem?

I understand that I should be scaling features between (0, 1) before feeding them into a neural network. However, what happens if future data could be larger than my current training data? For ...
3
votes
2answers
140 views

How to scale outputs from AutoEncoder from multiple models?

I have a problem for which I have not been able to find any answers in my search so far. BACKGROUND I am working on an anomaly detection problem on machines utilising an auto-encoder. I am building ...
2
votes
1answer
273 views

MinMaxScaler when LSTM predictions fall outside of training range?

I am using MinMaxScaler on my training set and applying the transformations to my test set and inverse_transform to my model’s outputs. If this were, say, a stock prediction problem, my training set ...
0
votes
1answer
121 views

Clustering, Mixed Data Set with Ordinal and Nominal Scale Data

After reading a bit how categorical data can be considered in clustering, I came to the conclusion that most of the post do not make distinction between nominal scale data e.g. colour: red, green, ...
1
vote
1answer
66 views

Scaling features separately?

I have some features which are in the thousands, which I scale to the max values of these. This solves the general scaling problems, as well as preserves an important absolute value relationship ...
1
vote
0answers
63 views

Data normalization of count data for neural networks

I have a sparse matrix of count data that I'm using as input to a neural network. I know, usually, the input data should be normalized (e.g. via min-max scaling, $z$-score standardization, etc.). But ...
6
votes
2answers
137 views

Best way to scale across different datasets

I have come across a peculiar situation when preprocessing data. Let's say I have a dataset A. I split the dataset into A_train ...
0
votes
1answer
74 views

Liner regression and feature scaling

Below are few questions where I unable to find out where I am wrong. I added screen shot of image and explanations of the each options that I am understanding. Questions are purely discussion based ...
0
votes
1answer
290 views

If we are using batch normalization as the first layer, can we forego standard scaling of inputs?

It is common practice to use the standard scaler on the inputs before feeding it to a deep learning architecture. I was wondering whether it is necessary if the first layer is a batch normalization ...
2
votes
1answer
101 views

Combining scaling, dimensionality reduction, prediction using sklearn pipeline

I would like to use a sklearn pipeline doing this : ( - ) scale the data ( StandardScaler ) ( - ) reduce dimensionality ( PCA ) ( - ) make a prediction with GradientBoostingRegressor() and ...
1
vote
0answers
30 views

Why is it necessary in batch normalization to multiply and add a parameter to the result?

How do we decide on which layer we want to add batch normalization. So if we have chosen a layer to apply batch norm to then why don't just normalize it why are we multiplying and scaling it by some ...
0
votes
1answer
34 views

Suggestion on Preprocessing dataset

I am trying to preprocess my dataset and needs some suggestion on it. The training data shape is : (166573, 14) The distribution of features : As you can see, only the first 4 columns go to ...
6
votes
3answers
343 views

How to give a higher importance to certain features in a (k-means) clustering model?

I am clustering data with numeric and categorical variables. To process the categorical variables for the cluster model, I create dummy variables. However, I feel like this results in a higher ...