Discovering the ML world with sklearn, I'm testing a large panel of models onto my dataset. This is for learning purpose but also for work so I want the final model to be as accurate as possible, while I can progress in my understanding of ML.

I've separated my dataset (16k rows) into 80% training and 20% testing, and I'm testing at least KNN, Logistic, DecisionTree, RandomForest, NaivesBayes and maybe SVC (if my computer can handle it), with some Bagging and Boosting when I find out how to.

My training sample comes in 4 ways: 2 sets of features (95 and 11), standartised (with a StandardScaler) or not.

My outcome is binary and I'm using a custom scorer "amelioration" which maximalise the number of positives for the 30th percentile (easier to get it with the code at the end of my post), along with specificity and roc_auc.

For each dataset, I cross validate (stratified and shuffled with a random state, 5 folds, with repetition when possible) all hyperparameters I find relevant and repeat this for each training sample.

For each crossvalidation, I refit using my scorer so I can compare the results amongst models and datasets. For models I've tested so far, my mean_test_amelioration range from +42% to +114%.

Finally, I'll measure the performances of the selected model on the testing sample and report results.

I guess this can feel quite cumbersome to a pro (and I'm probably building a tank to kill a fly), but I've already learned so much in this manner.

I'm only comparing all these models on the mean_test_amelioration and do not take into account standard error (for instance). Could this lead to overfitting so that my final model won't generalize well? If yes, how could I take variability into account ?

Any educationnal link is also very welcome.

PS: As this could be relevant, here is my custom scorer code:

def get_amelioration(y_true, y_pred, **kwargs):
       If I select 30% of my sample with this algorithm, I will have 
       `amelioration`% more positives in my selection than without

       :use as: make_scorer(get_amelioration, needs_proba=True, N=30)

    N = kwargs.pop('N', False)
    if kwargs: raise TypeError('Unexpected **kwargs: %r' % kwargs)

    decisions = (y_pred > np.percentile(y_pred, 100-N)).astype(int)

    tn, fp, fn, tp = metrics.confusion_matrix(y_true, decisions).ravel()
    v = (fp+tp)/(tn+fp+fn+tp)
    r = tp/(fp+tp)
    r_base = np.mean(y_true) #around 15% in my sample, expected to be stable
    amelioration = 100*(r/r_base-1)
#    print("N=%i, v=%0.3f, amelioration=%0.3f" %(N,v, amelioration))
    if v<0.75*N/100: return 0
    return amelioration

1 Answer 1


It is possible, however, this is usually not of great concern. If you would like to take into account the standard error, one possibility is the following:

  1. You compute the standard error of the best performing model (using your amelioration metric)
  2. Consider all models that perform within one standard error (or less if you prefer)
  3. Select the model with the simplest decision boundary

This is a heuristic(!) based on Occam's razor that simpler models tend to generalize better. However, keep in mind that for complicated problems simpler models are probably just that: simpler. They don't have to be more accurate.

Alternatively, instead of 3. you can select the model with the smallest standard error, if your main goal is to limit your exposure to this kind of variability.

Also keep in mind that there is No Unbiased Estimator of the Variance of K-Fold Cross-Validation, so this method brings with it its own uncertainty.

In general, I would just keep an eye on the standard error and if it's not too concerning just go with the numerically best-performing model.

  • $\begingroup$ Very interesting and intelligent concept thanks! Is it yours or is there something more to read anywhere? I'm not sure I understood the biased variance estimation caveat though. In a 5-fold CV, I would have 5 different and independant amelioration scores. This is the variance of these 5 values you are talking about? $\endgroup$ May 3, 2019 at 18:05
  • 1
    $\begingroup$ Yes. The standard error itself is subject to the bias-variance tradeoff. Not 100% sure where it originated but it usually known as the "one standard error rule". Tibshirani, for example, discusses it as a method to select the regularization parameter in the Lasso. It is briefly mentioned in ESL on pages 61 and 244. Here are a few more references. $\endgroup$
    – oW_
    May 3, 2019 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.