I have the following case:

Training data in the form of x, y coordinates on different frames (from a video). Based on this I computed some features, using only the training data and labels. A model is being trained with GridSearchCV.

It's a multiclass classification problem. There are too many coordinates available, some of which are not used to make features. There are also categorical features. The data is not balanced.

My pipeline consists of a preprocessor, that scales the numerical features with StandardScaler() and encodes categorical features with OneHotEncoder (actually those features are already encoded, but I didn't find another way to neglect those while scaling)

By applying SelectFromModel I reduce the number of features usually with threshold of '1.1*mean'

I am using stratified_group_k_fold from Kaggle (https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation) My data consists of different people, so I set the different people as groups. It seems that no matter which model I train, my CV score is on average 0.14 higher than the test score and never less than 0.12 higher.

Here's a snippet of my code.

# number of features
num_features = X_tr.shape[1]

# number of categorical features
num_categorical = int(np.sum([np.array_equal(X_tr[:,i], X_tr[:,i].astype(bool)) for i in range(X_tr.shape[1])])) 

#start index of categorical features
offset_cat = num_features-num_categorical

numeric_features = slice(0, offset_cat, 1)
numeric_transformer = Pipeline(steps=[('scaler', StandardScaler())])

onehot_features = slice(offset_cat, num_features, 1)
cat_list = [[0,1] for i in range(num_categorical)]

onehot_transformer = Pipeline(steps=[('onehot', OneHotEncoder(categories=cat_list))])

preprocessor = ColumnTransformer(transformers=[
    ('num', numeric_transformer, numeric_features),
    ('onehot', onehot_transformer, onehot_features)
from sklearn.svm import SVC
from sklearn.feature_selection import SelectFromModel

scorer = make_scorer(custom_loss_function, greater_is_better=True, needs_proba=True)
cv = stratified_group_k_fold(X_tr, y_tr, all_people, k=5, seed=42)

pipe = Pipeline(steps=[('preprocessor', preprocessor),
                       ('select', SelectFromModel(SVC(kernel='linear'), threshold='1.1*mean')),
                       ('clf', SVC(probability=True, random_state=42))])

param_grid = {
    'clf__C': Cs,
    'clf__gamma': gammas,
    'clf__random_state': [38]

search = GridSearchCV(pipe, 
                      iid=True, n_jobs=-1, 

search.fit(X_tr, y_tr, groups=all_people)

best_params = search.best_params_
cv_results = search.cv_results_

for k in param_grid:
    print(k, '=', best_params[k])
print('\n' + 
      'Training set score (map@3):' + str(search.score(X_tr, y_tr)))
print("Cross validation score:   ", search.best_score_)

clf__C = 12.50075

clf__gamma = 0.001

clf__random_state = 38

Training set score (map@3):0.9586247086247086

Cross validation score: 0.7175901022054868

While the test score on Kaggle is only 0.5713 I want to know if I'm doing something wrong here that could cause leakage. Specifically if there is something wrong with the stratified_group_k_fold because it doesn't seem to matter if I use this or just normal 5 fold cv by setting cv=5 in GridSearchCV

From the feedback I got, I know this gap could be closed. Unfortunately I only find ways to increase leakage instead of decreasing it. For example I tried removing outliers based on z-statistics of the features columns, but that only increased my CV-score and not the test score.

Main question:

Why is there leakage? Am I doing something wrong with Stratified Group K fold?

Side questions:

Is it normal for the CV score to be that much higher? What else could cause leakage in this case? Why does removing outliers increase leakage? Is there something I can do better?

Thanks for any help!

  • $\begingroup$ I don't think there is a leakage as far as I read your code. Do you happen to see other people's kernel that have test performance similar as CV? CV not matching with test performance is very common in kaggle by the way $\endgroup$ – Yohanes Alfredo Nov 21 '19 at 11:28
  • $\begingroup$ As @YohanesAlfredo says i don't think there is any "leakage" per se. Maybe you are referring to something else as "leakage". In machine learning, it usually means when you accidentally shared information between the test and training data-sets. Maybe the difference between the CV-score and expected score you are talking about is because you are using a CV estimate on your training set, while the test-score is much different. This could be because your test samples greatly differ from the ones in your train set or maybe the features + the classifier deployed cannot generalize well enough. $\endgroup$ – Bogas Nov 21 '19 at 12:58
  • $\begingroup$ Thanks for your replies! I'm glad to hear there's nothing causing leakage in the real sense of the word. I can not look at other people's kernels but since it's a private competition, the hosts told me they expect the gap between CV and test score to be at most 0.1. I will look into making better features and better models and hopefully get better results. $\endgroup$ – DataFace Nov 22 '19 at 5:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.