1
$\begingroup$

Which measurement(s) should one choose to compare two regression models?

After modifying a learning algorithm(specifically, a regression algorithm, let's call it M1) to generate another learning algorithm M2, how to validate if the above modification is efficient?

here is what I did(with 10-fold cross-validation)

I choose MSE as the only measurement, at each run, for M1 and M2, calculate the MSE of both the training and testing set.

And the result shows that:

  • average MSE of the training set of 10 runs: M2 < M1
  • average MSE of the testing set of 10 runs: M2 < M1

Question:

according to the above list, can we draw a conclusion that M2 is better than M1? thus, the modification of algorithm M1 is efficient(at least on this dataset)?

Or:

Did I miss some other important measurements? Is there a rule of thumb of comparing two regression models?

$\endgroup$
  • 1
    $\begingroup$ As per your results, on test set, MSE for M2 is lower. So yes, we can draw a conclusion that M2 is better than M1. You can also try comparing MSE with cross validation other than 10 fold (9 fold, 12 fold etc) but don't go too far than 10. If you get same results, your conclusion is correct. $\endgroup$ – Ankit Seth Apr 2 '18 at 8:26
  • $\begingroup$ @AnkitSeth Thanks for the help, but is it necessary to try some other k-fold cross-validations(k != 10)? $\endgroup$ – xtluo Apr 4 '18 at 1:35
  • $\begingroup$ I am saying this because k = 10 is not standard value for doing CV. $\endgroup$ – Ankit Seth Apr 4 '18 at 13:36
1
$\begingroup$

There are two things to consider:

  • Sampling bias
  • Metric

The sampling bias problem is that your test set is likely not the complete set of things you're interested in. So, no, you can't simply check MSE_1 < MSE_2 and conclude it is always the case when it's "just" for your dataset the case. This is what significance tests are for. (Although this kind of reasoning is super common in machine learning and I did it myself already 🙈)

Then the question if the metric is the correct one for your application. Typical choices are: MSE, mean absolute error, custom cost functions

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks for the help. About the sampling bias you mentioned, I think the issue "The sampling bias problem is that your test set is likely not the complete set of things you're interested in" occurs in the conventional validation(70% training, 30% testing), but not with k-fold cross-validation. So yes, we can draw a conclusion that M2 is better than M1 if the MSE of M2 is smaller than the MSE of M1 produced by 10-fold cross-validation. $\endgroup$ – xtluo Apr 4 '18 at 1:46
  • $\begingroup$ No, it always happens when you use a finite subset of an infinite number of possible inputs. $\endgroup$ – Martin Thoma Apr 4 '18 at 13:14
0
$\begingroup$

As a measurement perspective, according to me it's ok. But it is also possible that the set of hyper-parameters chosen for M1 is not the efficient one. And if you change and set correct hyper-param, possibly you may get different calculations altogether.

I think we should consider that measurement as well.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.