Questions tagged [feature-scaling]

Feature scaling is a data pre-processing step where the range of variable values is standardized. Standardization of datasets is a common requirement for many machine learning algorithms. Popular feature scaling types include scaling the data to have zero mean and unit variance, and scaling the data between a given minimum and maximum value.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Can I perform Verification and validation checks on datasets like AndroPRAguard, Drebin(contain malware and benign mobile apps)?

Verification and validation checks for data: Verification of data: - Visual Checks: It checks data visually. Double Entry Check: It checks duplication of data in the database. Validation of data: ...
0
votes
0answers
8 views

Target Encoding and Feature Scaling

I am using Support Vector Classification which performs well when we have done Feature Scaling, however, I am using Target Encoding on my categorical variables. Is it advisable to do Feature Scaling ...
0
votes
0answers
33 views

How to correctly manage predictions when the inputs are outbound the original scaling range?

I have a neural network for a regression problem that was trained using MinMaxScaler(0,1) for features and I have two questions with this. I often find that scaling the output (or target variable) ...
0
votes
0answers
15 views

HR Machine Learning: Treating/ Standardizing Part Time Employee Sums To Their Full Time Equivalents in Attrition Modeling

My data set consists of a subset of employees. Each employee has general HR information (typical standard hours, department, site, etc) along with punch card data which gives a clear picture of the ...
-1
votes
1answer
19 views

Understanding MSE,R2 scores wrt different scaling methods and non intutive results

EDIT: Added Code and updated the metric values as my code changed If I have the Income Statements of all the companies currently trading in the US, I would like to predict the gross profit. I was ...
2
votes
1answer
28 views

Overfitting with sklearn pipeline - reasons why?

So.... I've been playing around with this for FAR TOOO LONG now and I really need some advice. Most people on kaggle concat training and testing set TOGETHER and then pre scale the data, this seems ...
0
votes
0answers
11 views

transformation and standardization. what will be the order?

I have a dataset with all positive numeric values for a classification problem. out of 8 columns 4 columns have skewed distribution. What will be the ideal order to follow ? ...
0
votes
0answers
47 views

Normalizing Feature/Label with Negative Values

I am creating a neural network using tensorflow that predicts the energy consumption of a vehicle. Originally, I planned on normalizing all of the features from 0 to 1 using the scikit-learn object ...
2
votes
1answer
20 views

Standardization of variables in log-scale

I have a doubt regarding standardization, I have to use a multivariate regression and one of the variables is in log-scale. Is using standardization enough to re-scale the log variable, or is it ...
1
vote
3answers
91 views

How to handle features with very broad range

I have a long list of continuous values like in the image below: The plot looks like this: How to handle such features? If I train the model with this, the model will not have the best precision, ...
1
vote
3answers
81 views

why to use Scaler.fit only on x_train and not on x_test for normalizing value using MinMaxScaler?

while normalising the data everone is saying that we need to fit only on x_train and not on x_test ? why is that we should not fit x_test ? if we should not fit the scaler on x_test then why we need ...
4
votes
2answers
73 views

Why Decision Tree Classifier is not working with categorical value?

I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx. I am trying to model a Decision Tree Classifier as part of an ...
0
votes
1answer
26 views

Does Sklean's SGDClassifier automatically standardize the training data when regularization is turned on?

Generally speaking--it is best to apply standarizaton (z-scoring the training data) prior to regularization. Does sklearn.linear_model.SGDClassifier automatically standardize the training data or not ...
1
vote
1answer
33 views

One scaler for all features or one scaler per feature?

I have a time series with more than 30 features. For preprocessing with scikit learn do you usually use one scaler per feature or one scaler for all features that should be standardized/normalized?
1
vote
2answers
75 views

why does making the target variable normally distributed helps?

while working on some regression problems I have found that if the target variable is skewed, making it normally distributed(using transformations) almost always helps. Why is that? Should we also ...
11
votes
6answers
245 views

When should I NOT scale features

Feature scaling can be crucially necessary when using distance-, variance- or gradient-based methods (KNN, PCA, neural networks...), because depending on the case, it can improve the quality of ...
0
votes
1answer
56 views

Feature transformation possible at selected features or only at all?

I want to cluster. I have different features for that. Some features have a very small value range (from 0 to 0.8) and some have a very large value range (from 0 to 5 million). I want to use the ...
2
votes
2answers
76 views

Does Feature Normalization affect Gradient Descent | Linear Regression

am new to datascience and i want to learn linear regression so i coded linear regression from scratch and performed gradient descent to find the best $w_\theta$ and $b_\theta$ values using a tutorial. ...
0
votes
1answer
37 views

How to scale a variable when not knowing the maximum

I have a dataset with different features where some of them are not categorical, so they need to be scaled or normalized (especially the target). However, normalizing between 0-1 for instance means ...
0
votes
2answers
108 views

how to use standardization / standardscaler() for train and test?

At the moment I perform the following: ...
2
votes
1answer
37 views

Standardscaler() not standardscaling?

I have following pipeline: estimators = [] estimators.append(('standardize', StandardScaler())) prepare_data = Pipeline(estimators) Originally, the data looks ...
0
votes
0answers
12 views

Standardizing in each fold - Learning Curve

Problem Description Hello, I have a classification problem and I want to perform cross validation (with hyper parameter tuning) in order to evaluate the generalization of my models. Basically the ...
2
votes
1answer
55 views

When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?

I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
0
votes
0answers
9 views

Applying different feature scaling to different features in the same dataset

I am currently studying a course about big data and data mining, so I am new to this field. We were tasked to make a model using a training dataset of 60 columns and 68 rows, and then use the test ...
0
votes
1answer
90 views

Correcting for one of multiple strong batch effects in a dataset

I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it ...
0
votes
0answers
19 views

Do not scale Hog features?

when I train LinearSVC with the Hog features extracted from the Fashion-MNIST dataset then I get better results if I don't use StandardScaler before training than I use it. ...
1
vote
0answers
15 views

Variable importance of Numerical features in Classification Model - Random Forest Classifier

I have few numeric features in my model. Out of 25 features, I have 7-8 numeric features in my model. One thing I observed is model gives more weightage to numerical feature compare to categorical ...
2
votes
2answers
64 views

Categorical features preprocessing for clustering

Can anyone tell suggest the best practice for clustering data with mixtured features (both with categorical and continuous). I am struggling with a problem; I realized that for all metrics algorithms ...
1
vote
1answer
127 views

Should scaling be done for mixed data (categorical and numerical)?

My dataset contains 13 attributes consisting of 10 Numerical and 3 Categorical attributes and Target. It has 180 observations ...
1
vote
0answers
14 views

can i scale features within each input catgory differently

i am trying to predict laptop prices from previous sales. based on a text description of the laptop condition and the sale price. the affect of the condition on the final price is going to be ...
1
vote
1answer
32 views

very important features, but rarely input

The features are of capital gain and capital loss, but very small amounts of people have one or the other. As far as I can see it may not signify any great difference, though I'm not sure how I would ...
0
votes
0answers
9 views

Scale a column with respect to the deviation in another

I have a dataset consisting of 644 features and the temperature that the features where captured at. I know that the temperature will effect the value of some of the features. Is it possible to scale ...
0
votes
2answers
363 views

Should we denormalize our data after normalization?

If we use sklearn library's preprocessing.normalie() function to normalize our data before learning, like this: ...
0
votes
0answers
61 views

What is the difference between row-wise and column-wise Z-score normalization?

I have a data set, each row represents a movie name, each column is a feature (such as genres), I want to perform cosine similarity to find out the similarity between each movie, before that I need to ...
2
votes
2answers
46 views

Training vs test data set for supervised learning in real life scenario

In the tutorials, I have noticed only similar data has been used with models training and prediction. I was wondering how cases where you can't find training data that is similar to your final use ...
1
vote
0answers
31 views

Python is reading my data with NANS and Infs, but they don't have any

I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
0
votes
1answer
444 views

Clustering with geolocation (lat/long pairs) attributes

I am trying to cluster customer behavior based on where they shop given by lat/long pairs. I also have other numeric attributes such as volume, average amount spent, etc. I am considering using ...
0
votes
0answers
30 views

What happens if you add a constant value to all input data points to neural networks?

I have a somewhat basic question about neural networks. What would be the effect on the performance of a neural network if we add a constant value to all data points? For example, suppose you have ...
1
vote
1answer
72 views

How to transform stock data for LSTM-based neural network

I am trying to classify stock returns using an LSTM-based neural network. I would like to use closing price and volume as features (see below), but am unsure of whether I need to transform these (e.g....
0
votes
2answers
67 views

Effect of skewness in data

I am preparing classification model. Many of numeric variables are positives skewed. Should I change a distribution of variables to be more Gaussian?
1
vote
1answer
749 views

Feature engineering for categorical variables

I have some categorical variables in my dataset for a regression problem. 1) One of the variable can take 3 values (Girls, Boys, Girls&Boys). Converting it into one-hot encoding or binary ...
1
vote
0answers
21 views

Transformation of non-categorical discrete feature

Goal: Predict a performance score of a place of interest in a given city based on (amongst others), the number of restaurants within 200m. $\\$ Dataset: $D$ with a feature $x$ indicating the $\...
0
votes
1answer
200 views

How to scale exponential data for a regression problem?

I understand that I should be scaling features between (0, 1) before feeding them into a neural network. However, what happens if future data could be larger than my current training data? For ...
3
votes
2answers
166 views

How to scale outputs from AutoEncoder from multiple models?

I have a problem for which I have not been able to find any answers in my search so far. BACKGROUND I am working on an anomaly detection problem on machines utilising an auto-encoder. I am building ...
2
votes
1answer
330 views

MinMaxScaler when LSTM predictions fall outside of training range?

I am using MinMaxScaler on my training set and applying the transformations to my test set and inverse_transform to my model’s outputs. If this were, say, a stock prediction problem, my training set ...
0
votes
1answer
150 views

Clustering, Mixed Data Set with Ordinal and Nominal Scale Data

After reading a bit how categorical data can be considered in clustering, I came to the conclusion that most of the post do not make distinction between nominal scale data e.g. colour: red, green, ...
1
vote
1answer
89 views

Scaling features separately?

I have some features which are in the thousands, which I scale to the max values of these. This solves the general scaling problems, as well as preserves an important absolute value relationship ...
1
vote
0answers
74 views

Data normalization of count data for neural networks

I have a sparse matrix of count data that I'm using as input to a neural network. I know, usually, the input data should be normalized (e.g. via min-max scaling, $z$-score standardization, etc.). But ...
6
votes
2answers
191 views

Best way to scale across different datasets

I have come across a peculiar situation when preprocessing data. Let's say I have a dataset A. I split the dataset into A_train ...
0
votes
1answer
79 views

Liner regression and feature scaling

Below are few questions where I unable to find out where I am wrong. I added screen shot of image and explanations of the each options that I am understanding. Questions are purely discussion based ...