Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [feature-scaling]

Feature scaling is a data pre-processing step where the range of variable values is standardized. Standardization of datasets is a common requirement for many machine learning algorithms. Popular feature scaling types include scaling the data to have zero mean and unit variance, and scaling the data between a given minimum and maximum value.

0
votes
1answer
9 views

LSTM for prediction of next location step - help with standardization

I have a few questions regarding the topic and I hope someone might have experience with any of them. What I am trying to do is train an LSTM network, whose input is a sequence of N steps in a XYZ ...
2
votes
0answers
20 views

How to handle preprocessing (StandardScaler, LabelEncoder) when using data generator to train?

So, I have a dataset that is too big to load into memory all at once. Therefore I want to use a generator to load batches of data to train on. In this scenario, how do I go about performing scaling ...
0
votes
2answers
33 views

Should I scale my features?

I have a dataset that looks something like this; ...
0
votes
1answer
14 views

Dealing with normalized regression output

I have a regression model that is trained on a bunch of features and normalized targets so naturally when I use the model to predict on a new input, the output is also normalized (well not normalized ...
0
votes
1answer
29 views

Scaling label encoded values for Linear Algorithms

I have encoded categorical variables to numerical values. As we know that for feeding values to Linear Algorithms like SVM or KNN, we scale the values for columns having large variations. I have ...
2
votes
0answers
41 views

When should I use StandardScaler and when MinMaxScaler?

I have a feature vector with One-Hot-Encoded features and with continous features. How can I decide now, which data I shall scale with StandardScaler and which data scale with MinMaxScaler? I think I ...
0
votes
0answers
13 views

Plotting variable importance and cutoff values for variables

I am training a dataset which is normalized using the scale function in R. Then I train the data with cross-validation. I have a test set to test my prediction. I also normalize the test data to get ...
0
votes
0answers
61 views

Standard Scaler drops accuracy significantly in Scala Spark

I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly. Without the accuracy is ~90% (on training ...
0
votes
0answers
14 views

Preprocessing imbalanced features for deep learning

I have an imbalanced dataset containing some equally imbalanced features where many observations are stacked at 0% (note the log scales in the histograms here). The top row shows one of these features ...
0
votes
0answers
6 views

How to handle NULLS in a column with scalar values (for neural networks)

I'm hoping this is the correct location for this question and apologies if not. I have a data table of customer information vs an outcome. Each row represents a product purchase, and the product ...
1
vote
1answer
30 views

Scaling values for LSTM

I have the following time series data set Each row is a unique Item, and each column shows the amount purchased per day. There are a total of 33 columns. I'm taking the first 32 columns(leaving out ...
0
votes
2answers
16 views

How to normalize just one feature by scikit-learn?

Wanna apply a specific scaler, say StandardScaler, on a specific feature, keeping other features intact. the dataset format is something like: [ [1, 0.2, 1000], [2, 0.1, 2400], [3, 0.9, 7620] ] I ...
1
vote
0answers
10 views

Reciprocal rescaling of product of two matrices

I read in many papers about product of two matrices being invariant to reciprocal rescalings. What exactly does this means ?
0
votes
0answers
23 views

inverse transform test data with LSTM shape problem

I'm trying to use a LSTM with two feature but when i make predictions are can inverse transform them back as i seem to have a dimension problem. ""cannot broadcast shape (761,10),(2),(761,10). Im ...
0
votes
0answers
23 views

Need Help with inverse transform test data with LSTM

I'm trying to use a LSTM with two feature but when i make predictions are can inverse transform them back as i seem to have a dimension problem. ""cannot broadcast shape (761,10),(2),(761,10). Im ...
0
votes
1answer
30 views

Combining different features as input to Neural Network

I use two different sources of information as input to my neural model. The model takes a word as input and produces a 1/0 output. I represent each word by using its word embedding (1024 dimensional ...
0
votes
0answers
13 views

Same feature with different representation

lets suppose that I have two different datasets (A and B), and I want to train a single machine learning model using common features between them. One of these features called "score". Let's say that ...
1
vote
1answer
42 views

Is it better to use a MinMax or a Log Return normalization to predict stock price movements?

I am trying to use a LSTM model to predict d+2 and d+3 closing prices. I am not sure whether I should normalize the data with a MixMax scaler (-1,+1) using the log return (P(n)-P(0))/P(0) for each ...
0
votes
1answer
39 views

How to normalize data of a different nature?

I am working a price prediction LTSM model for the stock market. I am using multiple features: Open, Close, High and I would like to add the Volume. The 3 first features are of the same nature but ...
0
votes
2answers
38 views

Scaling features in artificial neural networks

So it is a well known thing that it is a good idea to scale features/training samples in the training set, so that the values do not differ too much in the absolute sense. For example we want to train ...
2
votes
1answer
36 views

Does Orange scale the data automatically for the linear regression with Ridge regularization

I'm using the linear regression tool with the Ridge regularization. To use the Ridge regularization I have to scale the data first. Does Orange scale the data automatically? I can't find any ...
1
vote
0answers
21 views

How standardizing and/or log transformation affect prediction result in machine learning models

I recently ran an elastic net model on my data. My predictors are mostly skewed. I found my model perform slightly better when I standardize on log-transformed data than standardizing on original data....
1
vote
0answers
19 views

Normalize data with uneven groups?

I have a dataset with 3 independent variables [city, industry, amount] and wish to normalize the amount. But I wish to do it with respect to industry and city. Simply grouping by the city and industry ...
0
votes
1answer
156 views

MinMaxScaler returned values greater than one

Basically I was looking for a normalization function part of sklearn, which is useful later for logistic regression. Since I have negative values, I chose MinMaxScaler with like so: ...
7
votes
1answer
190 views

What are some situations when normalizing input data to zero mean, unit variance is not appropriate or not beneficial?

I have seen normalization of input data to zero mean, unit variance many times in machine learning. Is this a good practice to be done all the time or are there times when it is not appropriate or not ...
1
vote
1answer
22 views

Using historical label as a feature in my ML model?

I am working on a predictive model to predict change in the price of an asset (up, down, no change). The labeling is based on the derivative of the price and is exponentially smoothed with an alpha of ...
2
votes
2answers
31 views

The use of feature scaling in scikit learn

I'm studing machine learning from here and the course uses 'Scikit Learn' for regression - https://www.udemy.com/machinelearning/ I can see that for some training regression algorithms, the author ...
1
vote
1answer
19 views

How to deal with Optional Input

I'm from the vision world and only worked with pixels from 0-255, ignoring any side effects. My current problem is different, in the way that I cannot rely on the input data. What my problem is: I ...
0
votes
1answer
33 views

Input standartization for Deep Learning - Proper Scaling

Typically the input to neural network (NN) is transformed to have zero mean and 1 std. I wonder why std scale should be 1? What about other scales? 10? 100? Doesn't it make sense to provide NN with ...
0
votes
0answers
13 views

Does it make sense to preprocess (normalise or standardise) this data for GAN?

I'm working on a project where I have a dataset for a dynamical system (pendulum) containing a trajectory, energy cost and corresponding control actions (See below). I'm using a generative adversarial ...
0
votes
0answers
7 views

Handling Error for Continuous Features in a Content-Based Filtering Recommender System

I've got a content-based recommender that works... fine. I was fairly certain it was the right approach to take for this problem (matching established "users" with "items" that are virtually always ...
0
votes
0answers
47 views

Noise Resulting from Inverse-Scaling for a Machine Learning Problem

I have created a neural network and am predicting reasonable values for most of my data (the task is multi-variate time series forecasting). I scale my data before inputting it using scikit-learn's ...
3
votes
1answer
639 views

Data scaling before or after PCA

I have seen senior data scientists doing data scaling either before or after applying PCA. What is more right to do and why?
1
vote
2answers
27 views

How to standarize feature vector with data in different scales?

Let's suppose I have a dataset with numerical attributes of different types. Let's suppose I want to employ a Neural Network for supervised classification with that dataset. For that, I need to ...
1
vote
3answers
95 views

How to Normalize a feature [closed]

I have a feature that income of individual. It ranges from 10k to 116 Million. I have about 300k+ records. Clearly, I cannot use this feature as is as it will distort the model output and there are ...
1
vote
0answers
11 views

how to model the implication of a feature in classification

I have a binary classification problem in which one feature (let's call it X) has implication property, i.e. if the feature is 1, the output of classification must ...
2
votes
1answer
29 views

Should I rescale tfidf features?

I have a dataset which contains both text and numeric features. I have encoded the text ones using the TfidfVectorizer from sklearn. I would now like to apply logistic regression to the resulting ...
0
votes
0answers
35 views

scaling features after imputing missing values

I have a set of features with missing values. After imputing them by the median, I perform feature scaling (in Python/scikit i use preprocessing.scale and imputer). Now, there are many zeros which ...
1
vote
0answers
8 views

Choosing parameters for testing

I have a DL4J project DL4J Framework for a CNN for Classification and Scaling. (Separate projects). My question is that how can I tune my parameters to achieve my goal? Note that my question is ...
0
votes
0answers
20 views

Scatter Plot on Map within a Small Range (a university campus)

I am trying to get a scatter plot of points based on geolocation data superposed on a map. I have already went through several libraries like basemap or plotly which do similar stuffs. However, the ...
2
votes
1answer
381 views

Data scaling before PCA: how to deal with categorical values?

I have to apply PCA on a dataset, which contains both numerical and categorical values. In the preprocessing phase, I converted all the categorical values in numerical, so that the software can deal ...
2
votes
1answer
286 views

How to normalize a boolean feature for neural nets?

I have a feature that is boolean and I would like to feed it to a neural net as one of the inputs. I think in theory the best is to encode as false->0 and true->1 because 0 as an input will deactivate ...
9
votes
3answers
3k views

Zero Mean and Unit Variance

I'm studying Data Scaling, and in particular the Standardization method. I've understood the math behind it, but it's not clear to me why it's important to give the features zero mean and unit ...
3
votes
1answer
662 views

Should one hot vectors be scaled with numerical attributes

In the case of having a combination of categorical and numerical Attributes, I usually convert the categorical attributes to one hot vectors. My question is do I leave those vectors as is and scale ...
2
votes
0answers
27 views

How to weigh feature array

I have a feature array of around 4000 elements, extracted from one source. On this array I've extracted 7 more feature from other source and now I basically have a 4007 feature array from each data ...
7
votes
4answers
88 views

Prediction from data series with varying features

I'm looking into a problem where the data points have unequal features. Each instance represents the progression of an item throughout the system. A number of them have progressed to their end point,...
2
votes
1answer
307 views

Using z-score for neural network normalization

I've read many people use z-score to normalize their data for presenting to a neural net, and that all data should lie in a range (usually -1 to 1), but z-score can return results beyond those bounds. ...
1
vote
1answer
282 views

How to Normalize & Scale a Single Data Point

I do understand the concept of normalizing & scaling the training/test data; it does help with the converging of the cost function. It is a great helper for many of the machine learning algorithms....
0
votes
1answer
2k views

How to use the same minmaxscaler used on the training data with new data?

Im using the keras LSTM model to make prediction, and the code above is to scale the data: inputs are shaped like (n, 11, 1) and the label is 1D DailyDemand.py ...
0
votes
1answer
75 views

How to do Feature Scaling for these ranges [0,1] and [-1,1]?

I want to rescale the features of my data to be between [0,1] and [-1,1]? Is their a clear cut way that works every time for these ranges? I think the below equation works for [0,1] but when it is ...