I am now working with a Linear Regression for a time-series regression problem (I am sorry that I cannot say too much about the problem and feature vector due to NDA).
I scaled both the input values and target (X and y) with scikit-learn StandardScaler().
The reasoning of scaling both X and y because since it is a time series regression problem, we are using the previous y value (target value) as the feature (X) for the next datapoint.
Thus, there is a relation between y and X.
So, I thought that in order to maintain the dependence of X and y (the previous target value), I need to scale them both.
After training the model, I got a low RMSE on my validation and test sets. Note that, the validation and test sets contain the data with scaled/normalized X and y values.
However, when I tried to rescale (inverse_transform) the predictions, and evaluate the RMSE of the predictions against the true target value of the test set (the true target value is already rescaled as well), I got a high RMSE error.
My question is, is it a good practice to evaluate RMSE/accuracy of rescaled (inverse transformed) predictions against the true target values?