How to interpret metrics of a model after scaling the data

Question

I have a GradientBoostingRegressor from scikit-learn which I trained. Afterwards, I obviously would like to know how good the model is. So, on a non-scaled dataset I would just use the mean_squared_error function from scikit and it would output a certain value that made sense (regarding the dataset).

Now, I scaled (/ transform) the targets in my dataset using the scikit QuantileTransformer(output_distribution='uniform'). The target is now scaled between 0 - 1. This is fine during training etc.

After training the model, I ran the following code to get a few metrics:

test_pred = gb.predict(X_test)
mse_test = mean_squared_error(y_test, test_pred)
print("RMSE on Test:", np.sqrt(mse_test))
print("MSE on Test:", mse_test)

mae_test = mean_absolute_error(y_test, test_pred)
print("MAE on Test:", mae_test)

Because the target values are scaled, the output is something similar to this:

RMSE on Test: 0.23563730007705744
MSE on Test: 0.05552493718760521
MAE on Test: 0.19235478752773819

I assumed that I could get the 'actual' non-scaled metrics back by applying the QuantileTransformer.inverse_transform function to the output.

So then I got:

RMSE on Test: 2231.21330222
MSE on Test: 807.28588575
MAE on Test: 1888.23406628

Which doesn't seem very correct to me. Normally, the RMSE would be smaller than the MSE, but that isn't the case. If you get that the sqrt of a (MSE) value under 1. Also, the MAE should probably be smaller than the MSE.

My question is, how do you interpret those scaled metric values? Is the inverse_transform output correct? How do I get correct, non-scaled values for the metrics?

I'd appreciate some help on this.

Thanks.

Edit: The QuantileTransformer is only an example. The question also applies to the MinMaxScaler and other scalers in general.

Answer 1

I solved this issue with the help of @Franziska W. Thanks!

I currently reverse transform y_test and y_pred and then calculate the metrics as following:

# qt_y is a QuantileTransformer instance, gb is a GradientBoostingRegressor
y_pred = gb.predict(X_test)
inv_y_pred = qt_y.inverse_transform(y_pred.reshape(-1, 1))
inv_y_test = qt_y.inverse_transform(y_test.reshape(-1, 1))

mse_test = mean_squared_error(inv_y_test, inv_y_pred)
print("RMSE on Test:", np.sqrt(mse_test))
print("MSE on Test:", mse_test)

mae_test = mean_absolute_error(inv_y_test, inv_y_pred)
print("MAE on Test:", mae_test)

Answer 2

Were your rescaled MSE and RMSE appear larger at the end? Should they be equal to the values before applying inverse transform?