1
$\begingroup$

I have the following time series data set

enter image description here

Each row is a unique Item, and each column shows the amount purchased per day. There are a total of 33 columns.

I'm taking the first 32 columns(leaving out the last column, which will be my target) as my training set, and the last 32 rows (leaving out the first column) as my testing set

X_train = dataset[:, :-1] # taking all columns except the last column

y_train = dataset[:, -1:] # setting the last column to be the target

X_test = dataset[:, 1:] # taking all columns expect the first column

I'm going to feed X_train and y_train in my LSTM model, and use the model to perform prediction on X_test.

Now, I wish to performing minmax scaling on dataset before performing training, but I have some questions:

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(-1, 1))
scaled_dataset = scaler.fit_transform(dataset)

This has obvious data leakages, because the scaler is fitted with the test values.

I thought of creating 2 scalers, one for the training set, and one for the target

scaler_x = MinMaxScaler(feature_range=(-1, 1))
scaler_y = MinMaxScaler(feature_range=(-1, 1))
scaled_all_data = scaler_x.fit_transform(dataset[:, :-1])
scaled_y = scaler_y.fit_transform(dataset[:, -1:])

I'm not sure if that's the right approach. I've already tried searching for answers, but their situation is not quite like mine, or the questions are unanswered yet.

Any advice on how I should perform value scaling?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ When scaling features, you should scale the training data and store the min and max values. Then, the test set has to be scaled with those values, not with the max and min of the test set $\endgroup$
    – ignatius
    Commented Jan 3, 2019 at 7:22

1 Answer 1

Reset to default
-1
$\begingroup$

Why not scale the values as the very first step? Import your values, scale the values and then do your splits for training/test/etc.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ You cannot fit the scaler based on the whole dataset. Otherwise, features of the Test set will flow back to the Train set. You have to scale based on Train set ONLY, and then use the Train scaling parameters to process the Test set. That's the only way to actually assess a model's performance $\endgroup$
    – Leevo
    Commented Nov 28, 2019 at 11:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.