0
$\begingroup$

I'm working on an employee attrition predictive model using sklearn's GradientBoostingClassfier. I have 9,000 observations, which I split 50/50 for training and testing. I have another set of 1,200 observations that I use for a final validation. All 10,200 observations were obtained in similar fashion.

I carried out a grid search with 5-fold cross-validation in order to obtain a suitable set of hyper parameters. The results for my test set are good and very stable. However, there is a big drop off in performance when use my final validation data.

Results for the test set

->  Precision: 0.836 / Recall: 0.629 / Accuracy 0.874

Results for the final validation set

->  Precision: 0.149 / Recall: 0.725 / Accuracy 0.484

At first I thought this could be the caused by data leakage, but even after removing "suspicious" features, there is still a big drop off when comparing the test results with the final validation results.

Surely I'm doing something wrong, but I'm at a loss as to what exactly. Here are the relevant lines of code (nothing fancy):

> X = pd.read_csv('train_test.csv')
> y = X.pop('Target')
> X_final = pd.read_csv('final_validation.csv')
> y_final = X_final.pop('Target')

> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)

> gb = GradientBoostingClassifier(n_estimators=300, max_depth=5, learning_rate=0.2)
> gb_model = gb.fit(X_train, y_train)

> # test set
> y_pred = gb_model.predict(X_test)
> precision, recall, fscore, support = score(y_test, y_pred, average='binary')

> # final validation set
> y_hat = gb_model.predict(X_final)
> precision, recall, fscore, support = score(y_final, y_hat, average='binary')

Any thoughts?

$\endgroup$

1 Answer 1

0
$\begingroup$

I would first suggest trying to plot the results during training. How do your metrics (or at least the loss) vary over the training process for training and cross-validation datasets? The loss at each iteration is appended to your GBC object gb_model in the train_score_ attribute.

Normally, when there is such a big gap between training and test data, it indicates that you are overfitting to your training data, and that the model does not generalise well to unseen data. You could think about doing shuffling your data, in order to balance the training/validation/test datasets - [if you are looking at a time-series problem, you should be careful as to how you do this].

$\endgroup$
3
  • $\begingroup$ Thanks for your reply. As you suggested, I plotted the train_score_ values, and the error decreases sharply at first and more steadily thereafter. It looks like many other curves I've seen online. Sorry for the newbie question, but I thought overfitting was a concern on the testing data. Here I have a fairly large test set (50%) for which the results are good. It's only when I use another set that performance drops. Is it common to suffer from overfitting even when performance is good on a large test set? $\endgroup$
    – J.R.
    May 2, 2018 at 18:51
  • $\begingroup$ It depends how you use the info from your cross-val. Using it to improve your model (which inherently you must be!), in essence means you use the 9000 train/test samples just for training - allowing the model to overfit. Testing on 1200 for final out-of-sample accuracy, mean a 88:12 split. 12% being 1200/10200. This split is another parameter to play with and is data dependent. I generally go for 70:20:10 (train:val:test). You could try: (1) a 10 or 20 fold cross-validation, (2) increasing the capacity of your model with max_depth & n_estimators (3) trying other models other than GBC. $\endgroup$
    – n1k31t4
    May 2, 2018 at 19:34
  • $\begingroup$ I see now what I was doing incorrectly. For my final set I just took the last 1200 observations I had, but I had not shuffled my entire dataset prior to doing so. When shuffling the dataset, the performance is stable. Thanks again for all your pointers. $\endgroup$
    – J.R.
    May 2, 2018 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.