How to evaluate the "betterness" of competitive good models?

Lets say I could get good models (> 90% prediction rate) with e.g.:

  • LinearSVC
  • F-test based sklearn.feature_selection.f_regression
  • mutual info -based sklearn.feature_selection.mutual_info_regression

But since these treat e.g. the independence/dependence of features differently, particularly e.g. LinearSVC assumes "relatively independent" features, where as mutual info particularly measure dependence between variables, then

How can I compare these models to each other?

Tests? Knowledge of the data a priori? Something else?

  • 1
    $\begingroup$ How can I compare these models to each other? how about a test set to evaluate which one has a lower generalization error. $\endgroup$ – Fadi Bakoura May 11 '18 at 18:16

The typical approach would be to compare (cross-)validation performance of the models, at least if accuracy is the selection criterion.

Other criteria could be

  • simplicity/ease to interpret
  • how easy the models could be implemented and/or updated in a productive environment
| improve this answer | |

This is a very general question, and the answer is that it depends on what you're doing, what your application is.

What is the reason you're building the model in the first place? You want to evaluate the performance, as Fadi Bakoura states, with a generalization error on a test set, but what error measure you choose depends on what you're doing, how balanced your dataset is, whether you care more about false positives or false negatives, or a number of other concerns.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.