Questions tagged [feature-scaling]
Feature scaling is a data pre-processing step where the range of variable values is standardized. Standardization of datasets is a common requirement for many machine learning algorithms. Popular feature scaling types include scaling the data to have zero mean and unit variance, and scaling the data between a given minimum and maximum value.
262
questions
1
vote
1
answer
19
views
Out-of-Range Target Variable in Sequence-based Machine Learning Model
I'm encountering a scaling issue in a machine learning project. I'm predicting a target variable from an input sequence (and doing this for many). However, I've encountered a challenge where the ...
- 11
0
votes
0
answers
13
views
How to deal with a small extra cluster in a tabular data?
I am working on a high dimensional tabular dataset with 1600 features and 9440 rows. No matter how I select the features, when I try to project my data into a 2d or 3d graph using dimensionality ...
1
vote
1
answer
27
views
Scale dataset while preserving relative distributions between columns
I have a large dataset with 460 columns. The columns have names such as
'AppOpen_1day', 'AppOpen_2day', ...... 'AppOpen_15day',
'Dig_Pos_1day', 'Dig_Pos_2day', ...... 'DigPos_15day'
etc. Each column ...
- 111
0
votes
0
answers
44
views
K nearest neighbor with varying feature length
I'm trying to build a tool which predicts the elemental composition of some light source using its emission spectrum with the k nearest neighbor algorithm (I'm using the KNeighborsClassifier from ...
- 1
0
votes
0
answers
32
views
How to work with multiple feature types on autoencoder?
This is my first post here.
I am working on an adversarial autoencoder that receives different features, encodes them, and decodes them.
For instance, suppose you have a dataset from a large survey ...
0
votes
0
answers
9
views
Which Frameworks/Libs Best Support Integer-Based Features, Scaling, Training, etc?
Papers such as Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference have interested me in exploring integer-based data science. In particular, I'm thinking of ...
0
votes
0
answers
13
views
Best distance metric and estadarization method for clustering with percentages data
I'm studying access patterns to a facility with clustering.
My variables are percentages. For example, for each user, I have the percentage of access 'in time' versus late, or the percentage of using ...
- 101
1
vote
0
answers
14
views
Clustering Similar Articles Using Mixed Data: Seeking Advice and Validation
Question:
I'm working on a project where I need to cluster a dataset of articles based on various features, including text, numeric values, and categorical data. I've implemented a clustering approach ...
- 11
0
votes
2
answers
37
views
Scaling of categorical feautres
In the context of algorithms that consider the scales of features, I have a situation where some features are encoded using ordinal encoding, some features are binary, and some features are standard ...
- 51
0
votes
1
answer
32
views
Different scaling methods of different features results in a faux dependency between them
My dataset contains the following two features: "movie duration" (minutes) and "tv shows duration" (seasons). If a certain sample is of type "movie", it's duration will ...
- 1
0
votes
1
answer
13
views
feature engineering mechanism
why do we need to rescale some feature having large range I know we do it for faster rate of gradient descent ,but still how does rescaling works? and it doesn't break the model
and does rescaling ...
1
vote
4
answers
75
views
Are scalers or encoders supposed to be serialized along with trained models?
Consider the very basic example below:
...
1
vote
1
answer
25
views
0
votes
0
answers
7
views
Feature Selection using Statistical Testing on Features with Different Scales
If I want to select features based on ANOVA test for example, should I scale features to same/similar range so the results can be compared , or is it unnecessary?
- 51
0
votes
1
answer
35
views
Standard Scaling not resulting in good nn inputs
I am using standard scaling to transform my dataset for input to a NN. The normalized dataset is in the range +-0.04. I am getting different results in the NN when simply multiplying by a factor of ...
- 101
5
votes
2
answers
555
views
How to handle similarity search on mixed data types vectors?
I think this question is one that many beginners run into and I could not find a decent generic guide for it.
My issue is the following. I want to evaluate similarity of vectors which have mixed data ...
- 53
1
vote
1
answer
85
views
How to encode & scale IP addresses as input for ML models
Im currently working on an anomaly detection while making a transaction. As a part of the data that I extracted, I have the IP addresses of the indivduals who made the transaction. Since the IP ...
1
vote
1
answer
433
views
StandardScaler and MinMaxScaler vs RobustScaler
I've recently read that Standard Scaler functions best in situations where the distribution of the features are approximately normal.
MinMaxScaler works in a way that it preserves the features' ...
0
votes
0
answers
33
views
Logistic Regression, Standardization, Stationarity, Differencing
I am going to be using the logistic regression in which I will use L2 Regularization. I have these 4 rolling standard deviation variables. Here are the results of the Augmented Dickey-Fuller Test for ...
0
votes
0
answers
18
views
Scaling nominal vars K means
I had a discussion recently with a coworker. We are running a K means clustering algorithm. He said that when dummy variables are made into 1s and 0s,these columns must be scaled in a specific way. If ...
- 1
0
votes
0
answers
13
views
Clustering methods for text and image features
I want to build a recommender system with unlabeled data and used TF-IDF to extract text features from a given short description and VGG-16 to extract image features. I am looking for a way to combine ...
- 1
0
votes
1
answer
252
views
Combining text and image features with different scales
I have computed text features using [SBERT][1] and image features using VGG-16. The text features range from -1.58 to 1.58, whereas the image features range between 0 and 521. I would want to ...
- 1
0
votes
1
answer
59
views
How to treat categorical columns after ordinal encoding?
If encode three categorical variables like "bad", "normal", "good" into 0,1,2, after that can I treat them as numerical values? So can I perform on them MinMaxScaler or ...
0
votes
0
answers
27
views
Scaling/Normalization again
I would like to ask for normalization .. again.
I'm working on LSTM model having a 100x4 dataset used for timeseries prediction.
My question is on which set of data the scalar/normalizer should be run ...
- 101
1
vote
2
answers
382
views
problem with standardScaler
problem with standardScaler
hi
I'd like to scale one column in the titanic data set. I am using the following code segment.
for some reason df_scaled results an empty set.
how can I solve it? what is ...
- 31
1
vote
1
answer
63
views
Customer Segmentation with mixed data [closed]
I want to perform clustering. I am reading about this topic but I am totally confused. My dataset has 490 observations and it consists of numerical data (3 columns: Recency, Frequency, Monetary), ...
- 11
2
votes
0
answers
211
views
Is there a need to use scaling for Age attribute?
What is the good way working with 'Age' attribute? Don't touch it or should it be scaled?
Below photo shows my results 'Before' and 'After' standardization.
- 121
0
votes
0
answers
15
views
How to Forecast Stock Index against Global Indicators
My goal is predict the benchmark index of the Indian stock market with the help machine learning algorithm using of global market indices as mentioned below. Put simply, forecast whether tomorrow’s ...
0
votes
0
answers
24
views
Is it sensible to use conventional scaling methods to scale features that have meaningful (+/-)sign?
I'm working on a project where the input features contain velocities of objects. The sign in the velocity feature is meaningful i.e. negative velocity means an object moving to the left while positive ...
- 101
0
votes
1
answer
35
views
Success metric of database migration using row counts
Description
I have a problem where I'm tasked to successfully transform and repurpose data from one SQL server to another. Call the source $\text{src}$ and the target database $\text{tgt}$.
In order ...
- 101
1
vote
1
answer
91
views
Is it valid to use Spark's StandardScaler on sparse input?
While I know it's possible to use StandardScaler on a SparseVector column, I wonder now if this is a valid transformation. My reason is that the output (most likely) will not be sparse. For example, ...
- 13
1
vote
1
answer
208
views
Inverse Scaling Partitioned Data
I have scaled an original matrix A with sklearn's StandardScaler, resulting to a matrix S.
I then partitioned the result into ...
- 11
0
votes
2
answers
96
views
Understanding the relationship between two features
I am trying to solve classification task. Could you suggest me, if these two features are independent? the plot looks strange
0
votes
1
answer
656
views
Scaling the target variables for Neural Networks in Regression Problem
I am trying to implement a neural network on a regression problem. I have scaled the independent variables since this is a crucial step for neural networks. However, I see online that some people ...
- 221
0
votes
1
answer
210
views
Standardization after log transformation
I have a few question about log transformation and standardization.
First: Should I standardize my features after doing log transformation?
Second: I still do not understand, because when doing log ...
- 138
0
votes
1
answer
173
views
How to scale a subset of data with respect to the entire dataset
I am developing a financial time-series prediction model using sklearn using StandardScaler for scaling purposes. I train a model, and then use the model regularly ...
- 111
0
votes
2
answers
780
views
Should I split data into train/validation/test before feature scaling and feature selection or after?
I'm working on a project, I finished data preprocessing, and I found an article where it says that feature scaling and feature selection should be done after splitting data, some other articles say it ...
- 11
0
votes
1
answer
414
views
Feature scaling in Linear Regression
I always use Linearregression() class in sklearn library for creating a linear regression model. According to my understanding, we need feature scaling in linear ...
- 35
2
votes
1
answer
2k
views
Predict actual result after model trained with MinMaxScaler LinearRegression
I was doing the modeling on the House Pricing dataset. My target is to get the mse result and predict with the input variable
I have done the modeling, I'm doing the modeling with scaling the data ...
- 123
3
votes
1
answer
90
views
What justifies feature scaling?
Although I can understand the significance of feature scaling in some cases (e.g. when gradient descent is involved), I don't feel I understand the necessity of this process in general. But there a ...
- 181
2
votes
0
answers
30
views
When is scaling and centering important?
There are some models such as PCA or SVM where scaling and centering of training data is essential.
There are some models, mostly tree-based where scaling and centering is not required at all.
I don't ...
- 121
0
votes
1
answer
61
views
How to Scale target feature
How should I scale target feature? Should I use scaler as fit_transform on y_train, and just fit on y_test to avoid leaking data?
2
votes
1
answer
48
views
Choosing Right Optimiser and Data Scaling
The choice of optimiser and how data is scaled are both very important things in machine learning, yet they are not hyperparameters (as far as I am aware). It is also not necessarily obvious which ...
- 91
1
vote
2
answers
136
views
Ways of scaling scores on data without knowing possible maximum values
In my scenario, I have to process some input data and give a score based on what the processing phase outputs. The problem is that, in order to scale the score in a human-readable format I'd have to ...
1
vote
1
answer
382
views
Is test data required to be transformed by training data statistics?
I am using a dataset (from literature) to build an MLP and classify real-world samples (from wetlab experiment) using this MLP. The performance of MLP on the literature dataset are well enough. I am ...
0
votes
0
answers
125
views
Do I need to encode numerical variables like "year"?
I have a simple time-series dataset. it has a date-time feature column.
...
- 1
0
votes
1
answer
109
views
Why does log-transforming the target have a huge impact on MSE value?
I am doing linear regression using the Boston Housing data set, and the effect of applying $\log(y)$ has a huge impact on the MSE. Failing to do it gives MSE=34.94 ...
- 119
1
vote
1
answer
3k
views
Feature selection before or after scaling and splitting
Should feature scaling/standardization/normalization be done before or after feature selection, and before or after data splitting?
I am confused about the order in which the various pre-processing ...
- 119
0
votes
2
answers
95
views
Standardization in combination with scaling
Would it be ok to standardize all the features that exhibit normal distribution (with StandardScaler) and then re-scale all the features in the range 0-1 (with <...
- 119
2
votes
2
answers
248
views
Is it wise to always `StandardScaler()` features? [SOLVED]
My current investigations point to the sklearn.preprocessing.StandardScaler() not always being the right choice for certain types of feature extractions for neural ...
