Model comparison with CV using standard error

Question

Discovering the ML world with sklearn, I'm testing a large panel of models onto my dataset. This is for learning purpose but also for work so I want the final model to be as accurate as possible, while I can progress in my understanding of ML.

I've separated my dataset (16k rows) into 80% training and 20% testing, and I'm testing at least KNN, Logistic, DecisionTree, RandomForest, NaivesBayes and maybe SVC (if my computer can handle it), with some Bagging and Boosting when I find out how to.

My training sample comes in 4 ways: 2 sets of features (95 and 11), standartised (with a StandardScaler) or not.

My outcome is binary and I'm using a custom scorer "amelioration" which maximalise the number of positives for the 30th percentile (easier to get it with the code at the end of my post), along with specificity and roc_auc.

For each dataset, I cross validate (stratified and shuffled with a random state, 5 folds, with repetition when possible) all hyperparameters I find relevant and repeat this for each training sample.

For each crossvalidation, I refit using my scorer so I can compare the results amongst models and datasets. For models I've tested so far, my mean_test_amelioration range from +42% to +114%.

Finally, I'll measure the performances of the selected model on the testing sample and report results.

I guess this can feel quite cumbersome to a pro (and I'm probably building a tank to kill a fly), but I've already learned so much in this manner.

I'm only comparing all these models on the mean_test_amelioration and do not take into account standard error (for instance). Could this lead to overfitting so that my final model won't generalize well? If yes, how could I take variability into account ?

Any educationnal link is also very welcome.

PS: As this could be relevant, here is my custom scorer code:

def get_amelioration(y_true, y_pred, **kwargs):
    """
       If I select 30% of my sample with this algorithm, I will have 
       `amelioration`% more positives in my selection than without

       :use as: make_scorer(get_amelioration, needs_proba=True, N=30)
    """

    N = kwargs.pop('N', False)
    if kwargs: raise TypeError('Unexpected **kwargs: %r' % kwargs)

    decisions = (y_pred > np.percentile(y_pred, 100-N)).astype(int)

    tn, fp, fn, tp = metrics.confusion_matrix(y_true, decisions).ravel()
    v = (fp+tp)/(tn+fp+fn+tp)
    r = tp/(fp+tp)
    r_base = np.mean(y_true) #around 15% in my sample, expected to be stable
    amelioration = 100*(r/r_base-1)
#    print("N=%i, v=%0.3f, amelioration=%0.3f" %(N,v, amelioration))
    if v<0.75*N/100: return 0
    return amelioration

Answer 1

It is possible, however, this is usually not of great concern. If you would like to take into account the standard error, one possibility is the following:

1. You compute the standard error of the best performing model (using your amelioration metric)
2. Consider all models that perform within one standard error (or less if you prefer)
3. Select the model with the simplest decision boundary

This is a heuristic(!) based on Occam's razor that simpler models tend to generalize better. However, keep in mind that for complicated problems simpler models are probably just that: simpler. They don't have to be more accurate.

Alternatively, instead of 3. you can select the model with the smallest standard error, if your main goal is to limit your exposure to this kind of variability.

Also keep in mind that there is No Unbiased Estimator of the Variance of K-Fold Cross-Validation, so this method brings with it its own uncertainty.

In general, I would just keep an eye on the standard error and if it's not too concerning just go with the numerically best-performing model.