-1
$\begingroup$

I am using regression tree to predict target variable(continuous).

I've use one-hot encoding for all categorical features and applied standard scaler to all numerical features. After all that I train and test using regression tree and I get

train_MSE = 0
test_MSE = 0.11

given target variable ranges from [0,140], and mean of 60(Edited).

Just by looking at train_MSE I was worried that it is overfitted however test_MSE seems pretty good as well, can I simply interpret this result as "model is doing a good job"? I am worried that I might be missing something.

$\endgroup$
2
  • $\begingroup$ How was feature selection done? Is it possible that there was some information leakage at that stage? How many samples, how many numerical features, and how many one-hot features? $\endgroup$
    – C8H10N4O2
    Nov 16, 2020 at 16:51
  • $\begingroup$ I've removed features like "id", checked for multicolinearity and found none. There are 3 categorical features, 2 numerical features, and 2 ordinal features(year, week) . 3 categorical features + 2 ordinal features are one hot encoded. $\endgroup$
    – haneulkim
    Nov 16, 2020 at 21:35

1 Answer 1

1
$\begingroup$

MSE (as well as MAE) depends on the unit/scale of the entity being predicted. For example, if you measure your predictor variable in meters or centimeters will directly affect the MSE (low MSE when you use meters compared to centimeters).

One option you can consider is to look at the relative errors (errors divided by the true values). For example, relative RMSE or relative MAE.

$\endgroup$
8
  • $\begingroup$ Yes, that's why I gave info about target variable range. $\endgroup$
    – haneulkim
    Nov 16, 2020 at 23:59
  • 1
    $\begingroup$ Range alone doesn't have much information. If your data is uniformly distributed in the 0 to 140 range, then the MSE alone is fine. If your relative RMSE (or similar meassure) is also low for the test data, then that would be good enough and you don't have to worry about the data range or distribution. $\endgroup$ Nov 17, 2020 at 1:10
  • $\begingroup$ Range I am referring to is value range of target variable. Just like you said before it is important to know whether you are using m or cm, in my case since target value ranges from 0, 140 and mean of 60 therefore having MSE 0.11 indicates good performance since if target it 60 its prediction will be between 59.89 and 60.11 on average which I think is pretty close. Maybe knowing mean value will help as it indicates it is not uniform distribution. $\endgroup$
    – haneulkim
    Nov 17, 2020 at 7:04
  • 1
    $\begingroup$ If you know more about the target distribution, then MSE alone would be good enough. For example, if the the test data target distribution is a representative sample, and, say, it is like a normal distribution with a mean of 60, and not very high std, then this definitely is a pretty good model. $\endgroup$ Nov 17, 2020 at 10:52
  • 1
    $\begingroup$ Q1. It is not possible to say anything like that. Q2. Not clear if the model it fitted well or over fitted with the training data. If the test set is a good one, the this is probably a good model. $\endgroup$ Nov 17, 2020 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.