0
$\begingroup$

How do we evaluate the performance of a regression model with a certain RMSE given that a domain knowledge performance metric is not present?

Maybe MAPE is one way of comparing the performance of my model on my dataset with another model on another but a similar dataset.

Is there any way to evaluate the performance of my trained model given summary statistics and RMSE values?

To illustrate, here are the summary statistics of the target value:

Mean = 3.33
Median = 2
STD = 3.24
Max = 92 (no data error)
RMSE:1.97

My RMSE value is significantly lower than the standard deviation. should this indicate that my model is "good enough" ? I mean how can I be sure that my model does not underfit the data ?

$\endgroup$
5
  • $\begingroup$ Can you compare with a couple different models? That can be a benchmark, $\endgroup$ Mar 30, 2023 at 11:51
  • $\begingroup$ yes. but let's say I compared it with other models and I got a lower score such as 1.8. It is again me who does the modelling. I can not be the benchmark. can I? :) I'd suggest maybe a better data scientist would get a better score with better handling of the hyperparameters and feature engineering. $\endgroup$ Mar 30, 2023 at 11:54
  • $\begingroup$ If not you then who? Do you have other people that are willing to model this? $\endgroup$ Mar 30, 2023 at 11:55
  • $\begingroup$ I don't. that is why I am looking for a mathematical way of proving that my model is good enough. $\endgroup$ Mar 30, 2023 at 11:57
  • $\begingroup$ No such thing, mathematics won't help you here. $\endgroup$ Mar 30, 2023 at 11:59

1 Answer 1

0
$\begingroup$

Under parameterize the regression model by starting with a model that is just an intercept. Then perform forward selection, add coefficients one at a time to find a model that is not under-fit.

Model comparison is the best way to evaluate a model. Does one model perform better than another model?

$\endgroup$
2
  • $\begingroup$ That's almost surely going to overfit, so not a very useful metric. $\endgroup$ Mar 30, 2023 at 13:34
  • $\begingroup$ You are correct. I misread the question. I have revised my answer. $\endgroup$ Mar 30, 2023 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.