0
$\begingroup$

I have serious doubts concerning the features standardization done before the learning process of a multilayer perceptron.

I'm using python-3 and the scikit-learn package for the learning process and for the features normalization.

As suggested from the scikit-learn wiki (Tips on pratical use), I'm doing a features standardization with the Preprocessing module, which means that all my features will appear as standard normal specifications.

The problem is that in the showed example:

from sklearn.preprocessing import StandardScaler  
scaler = StandardScaler()  
# Don't cheat - fit only on training data
scaler.fit(X_train)  
X_train = scaler.transform(X_train)  
# apply same transformation to test data
X_test = scaler.transform(X_test)  

The standardization is done first on the training set, and then, after that, it's applied on the test set.

Which means, that the distributions parameters (mean, st.deviation) will be estimated only from the training set.

Now, because I need to save my multiple datasets in a database, on which I will perform the hold-out sampling for the training-test folds, I have done a preprocessing on all my instances at once, because I don't want to do that at run time execution.

It's clear that in this way the distributions parameters will change and the results will be different.

Now, Someone knows the differences between the two approaches? Are there issues doing the preprocessing at once on all the dataset?

$\endgroup$
1
$\begingroup$

Generally, preprocessing parameters are fit only on the training subset, because otherwise you could overfit your data and overestimate quality of your model on the test subset.

With feature standardization, however, overfitting is not so dangerous, so I assume you can preprocess all your dataset at once safely.

The best practice, however, is to do all the processing in one pipeline and apply cross-validation to the whole pipeline:

from sklearn.pipeline import make_pipeline
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score, GridSearchCV
scaler = StandardScaler()  
model = MLPClassifier()
pipeline = make_pipeline(scaler, model)
scores = cross_val_score(pipeline, X, y)

If you use scikit-learn cross-validation, incorporating pipelines would be simple.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.