Discovering the ML world with
sklearn, I'm testing a large panel of models onto my dataset. This is for learning purpose but also for work so I want the final model to be as accurate as possible, while I can progress in my understanding of ML.
I've separated my dataset (16k rows) into 80% training and 20% testing, and I'm testing at least
NaivesBayes and maybe
SVC (if my computer can handle it), with some
Boosting when I find out how to.
My training sample comes in 4 ways: 2 sets of features (95 and 11), standartised (with a
StandardScaler) or not.
My outcome is binary and I'm using a custom scorer "amelioration" which maximalise the number of positives for the 30th percentile (easier to get it with the code at the end of my post), along with
For each dataset, I cross validate (stratified and shuffled with a random state, 5 folds, with repetition when possible) all hyperparameters I find relevant and repeat this for each training sample.
For each crossvalidation, I refit using my scorer so I can compare the results amongst models and datasets. For models I've tested so far, my
mean_test_amelioration range from +42% to +114%.
Finally, I'll measure the performances of the selected model on the testing sample and report results.
I guess this can feel quite cumbersome to a pro (and I'm probably building a tank to kill a fly), but I've already learned so much in this manner.
I'm only comparing all these models on the
mean_test_amelioration and do not take into account standard error (for instance). Could this lead to overfitting so that my final model won't generalize well? If yes, how could I take variability into account ?
Any educationnal link is also very welcome.
PS: As this could be relevant, here is my custom scorer code:
def get_amelioration(y_true, y_pred, **kwargs): """ If I select 30% of my sample with this algorithm, I will have `amelioration`% more positives in my selection than without :use as: make_scorer(get_amelioration, needs_proba=True, N=30) """ N = kwargs.pop('N', False) if kwargs: raise TypeError('Unexpected **kwargs: %r' % kwargs) decisions = (y_pred > np.percentile(y_pred, 100-N)).astype(int) tn, fp, fn, tp = metrics.confusion_matrix(y_true, decisions).ravel() v = (fp+tp)/(tn+fp+fn+tp) r = tp/(fp+tp) r_base = np.mean(y_true) #around 15% in my sample, expected to be stable amelioration = 100*(r/r_base-1) # print("N=%i, v=%0.3f, amelioration=%0.3f" %(N,v, amelioration)) if v<0.75*N/100: return 0 return amelioration