# Why is there a big drop off in performance in my GBM?

I'm working on an employee attrition predictive model using sklearn's GradientBoostingClassfier. I have 9,000 observations, which I split 50/50 for training and testing. I have another set of 1,200 observations that I use for a final validation. All 10,200 observations were obtained in similar fashion.

I carried out a grid search with 5-fold cross-validation in order to obtain a suitable set of hyper parameters. The results for my test set are good and very stable. However, there is a big drop off in performance when use my final validation data.

Results for the test set

->  Precision: 0.836 / Recall: 0.629 / Accuracy 0.874


Results for the final validation set

->  Precision: 0.149 / Recall: 0.725 / Accuracy 0.484


At first I thought this could be the caused by data leakage, but even after removing "suspicious" features, there is still a big drop off when comparing the test results with the final validation results.

Surely I'm doing something wrong, but I'm at a loss as to what exactly. Here are the relevant lines of code (nothing fancy):

> X = pd.read_csv('train_test.csv')
> y = X.pop('Target')
> y_final = X_final.pop('Target')

> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)

> gb = GradientBoostingClassifier(n_estimators=300, max_depth=5, learning_rate=0.2)
> gb_model = gb.fit(X_train, y_train)

> # test set
> y_pred = gb_model.predict(X_test)
> precision, recall, fscore, support = score(y_test, y_pred, average='binary')

> # final validation set
> y_hat = gb_model.predict(X_final)
> precision, recall, fscore, support = score(y_final, y_hat, average='binary')


Any thoughts?

I would first suggest trying to plot the results during training. How do your metrics (or at least the loss) vary over the training process for training and cross-validation datasets? The loss at each iteration is appended to your GBC object gb_model in the train_score_ attribute.
• It depends how you use the info from your cross-val. Using it to improve your model (which inherently you must be!), in essence means you use the 9000 train/test samples just for training - allowing the model to overfit. Testing on 1200 for final out-of-sample accuracy, mean a 88:12 split. 12% being 1200/10200. This split is another parameter to play with and is data dependent. I generally go for 70:20:10 (train:val:test). You could try: (1) a 10 or 20 fold cross-validation, (2) increasing the capacity of your model with max_depth & n_estimators (3) trying other models other than GBC. May 2, 2018 at 19:34