(De-)Scaling/normalizing input and output data inside Keras model as layer

Question

I am building a 2-hidden layer MLP using Keras. I'm using a SciKit learn wrapper to be able to use the GridSearchCV functionality. My sample-size is limited, forcing me to use K-fold verification as well for trustworthy results.

However, it is to my understanding that for every iteration in the K-fold validation, input data should be scaled (and output descaled) only using the training data. This requires the scaling to be performed inside the Keras model.

In order to have understandable results, the output should than be transformed back (using previously found scaling parameters) in order to calculate the metrics.

Is it possible to

1. Z-score standardize my input data (X & Y) in a normalization layer (batchnormalization for example)
2. Transform the output layer back (before calculating metrics), using scaling parameters found in 1.

I've looked at the batchnormalization functionality in Keras, but the documentation mentions: "During training time, BatchNormalization.inverse and BatchNormalization.forward are not guaranteed to be inverses of each other because inverse(y) uses statistics of the current minibatch, while forward(x) uses running-average statistics accumulated from training." Which seems like this prevents the functionality from being used in this manner.

Does anybody have a known solution or function for this?

Answer 1

Let's start from X. As you use the sklearn-style interface, the natural choice is to use StandardScaler. You could use pipeline to integrate it in the grid search.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
pipeline = Pipeline([
   ('scaler', StandardScaler()),
   ('model', your_cool_model)])
param_grid = {...}
search = GridSearchCV(pipeline, param_grid, cv=5).fit(X, y)

That would do the trick.

Now about y. The chances are you don't need to scale target. The good discussion is in this question

If you are certain that targets should be scaled, the sklearn provides TransformedTargetRegressor just for the case

wrapped_model = TransformedTargetRegressor(
    regressor=model,
    transformer=StandardScaler())