Is it a good practice to evaluate model performance by comparing the metrics of rescaled (inverse transformed) predictions and true target values?

Question

I am now working with a Linear Regression for a time-series regression problem (I am sorry that I cannot say too much about the problem and feature vector due to NDA).

I scaled both the input values and target (X and y) with scikit-learn StandardScaler().
The reasoning of scaling both X and y because since it is a time series regression problem, we are using the previous y value (target value) as the feature (X) for the next datapoint.
Thus, there is a relation between y and X.
So, I thought that in order to maintain the dependence of X and y (the previous target value), I need to scale them both.

After training the model, I got a low RMSE on my validation and test sets. Note that, the validation and test sets contain the data with scaled/normalized X and y values.

However, when I tried to rescale (inverse_transform) the predictions, and evaluate the RMSE of the predictions against the true target value of the test set (the true target value is already rescaled as well), I got a high RMSE error.

My question is, is it a good practice to evaluate RMSE/accuracy of rescaled (inverse transformed) predictions against the true target values?

Answer 1

The error as a percentage of the true value in each case is probably the same - it just seems bigger because the standardised values are a lot smaller than the unstandardised ones (standardised values will be mostly between -1 and 1 and the unstandardised values will be whatever they are in your dataset).

Try measuring the fit of your linear model using Pearson’s correlation coefficient, checking how your predicted values correlate to the true values. This should be the same for both the scaled and unscaled cases and will give you a better idea of whether your model is good.

You could also plot your predicted values against the true values in a scatter plot so that you can see roughly what’s going on.