How to evaluate the "betterness" of competitive good models?

Question

How to evaluate the "betterness" of competitive good models?

Lets say I could get good models (> 90% prediction rate) with e.g.:

• LinearSVC
• F-test based sklearn.feature_selection.f_regression
• mutual info -based sklearn.feature_selection.mutual_info_regression

But since these treat e.g. the independence/dependence of features differently, particularly e.g. LinearSVC assumes "relatively independent" features, where as mutual info particularly measure dependence between variables, then

How can I compare these models to each other?

Tests? Knowledge of the data a priori? Something else?

Answer 1

The typical approach would be to compare (cross-)validation performance of the models, at least if accuracy is the selection criterion.

Other criteria could be

• simplicity/ease to interpret
• how easy the models could be implemented and/or updated in a productive environment

Answer 2

This is a very general question, and the answer is that it depends on what you're doing, what your application is.

What is the reason you're building the model in the first place? You want to evaluate the performance, as Fadi Bakoura states, with a generalization error on a test set, but what error measure you choose depends on what you're doing, how balanced your dataset is, whether you care more about false positives or false negatives, or a number of other concerns.