1
$\begingroup$

I have a GradientBoostingRegressor from scikit-learn which I trained. Afterwards, I obviously would like to know how good the model is. So, on a non-scaled dataset I would just use the mean_squared_error function from scikit and it would output a certain value that made sense (regarding the dataset).

Now, I scaled (/ transform) the targets in my dataset using the scikit QuantileTransformer(output_distribution='uniform'). The target is now scaled between 0 - 1. This is fine during training etc.

After training the model, I ran the following code to get a few metrics:

test_pred = gb.predict(X_test)
mse_test = mean_squared_error(y_test, test_pred)
print("RMSE on Test:", np.sqrt(mse_test))
print("MSE on Test:", mse_test)

mae_test = mean_absolute_error(y_test, test_pred)
print("MAE on Test:", mae_test)

Because the target values are scaled, the output is something similar to this:

RMSE on Test: 0.23563730007705744
MSE on Test: 0.05552493718760521
MAE on Test: 0.19235478752773819

I assumed that I could get the 'actual' non-scaled metrics back by applying the QuantileTransformer.inverse_transform function to the output.

So then I got:

RMSE on Test: 2231.21330222
MSE on Test: 807.28588575
MAE on Test: 1888.23406628

Which doesn't seem very correct to me. Normally, the RMSE would be smaller than the MSE, but that isn't the case. If you get that the sqrt of a (MSE) value under 1. Also, the MAE should probably be smaller than the MSE.

My question is, how do you interpret those scaled metric values? Is the inverse_transform output correct? How do I get correct, non-scaled values for the metrics?

I'd appreciate some help on this.

Thanks.

Edit: The QuantileTransformer is only an example. The question also applies to the MinMaxScaler and other scalers in general.

$\endgroup$
2
  • 1
    $\begingroup$ I would reverse transform y_test and y_pred and then calculate the metrics MSE, RMSE and MAE from these values. $\endgroup$ – Franziska W. May 4 '19 at 6:14
  • $\begingroup$ Thanks! That seems to work correctly indeed. $\endgroup$ – Brontosaurs May 6 '19 at 6:47
0
$\begingroup$

I solved this issue with the help of @Franziska W. Thanks!

I currently reverse transform y_test and y_pred and then calculate the metrics as following:

# qt_y is a QuantileTransformer instance, gb is a GradientBoostingRegressor
y_pred = gb.predict(X_test)
inv_y_pred = qt_y.inverse_transform(y_pred.reshape(-1, 1))
inv_y_test = qt_y.inverse_transform(y_test.reshape(-1, 1))

mse_test = mean_squared_error(inv_y_test, inv_y_pred)
print("RMSE on Test:", np.sqrt(mse_test))
print("MSE on Test:", mse_test)

mae_test = mean_absolute_error(inv_y_test, inv_y_pred)
print("MAE on Test:", mae_test)
$\endgroup$
0
$\begingroup$

Were your rescaled MSE and RMSE appear larger at the end? Should they be equal to the values before applying inverse transform?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.