I am attempting a classification project. I have split my data, 20000ish, into training and test sets. On the training sets I run a selection of classifieds including knn random forests and gbm . These give me about 20-30% accuracy at best. For each sample I generate probabities of each class and make a new model

Knn proba 1, knn proba 2.... Random forests proba 1 etc

On this I then run a random forest classifier which gives me a 90ish% accuracy against the test set.

Fantastic!... But when I use the model against new data the accuracy is very poor.

In part it feels like a case of overfitting but surely the test set should be poor as well

Why might the test data so good but new data so poor... What have I done wrong?

Thanks Chris

  • 1
    $\begingroup$ Have you overlapped test set data somehow, so the RF is being leaked the test data via one of the lower-level models? Please explain more about your data splitting strategy. $\endgroup$ Aug 16, 2015 at 20:43
  • $\begingroup$ So it turns out that Neil was right - in my code to split the data sets I had an error (or instead of an and) which meant that half my test data was included in the training set and thus over-fitting. $\endgroup$
    – Chris
    Sep 17, 2015 at 14:15

2 Answers 2


Yes I suspect you are overfitting. When you build your first stage of models (nearest neighbors, random forest, gradient boosting, etc...) is the process like this?

random_forest_probabilities = random_forest.predict_proba(train_data)

If so you are plugging in the same data you used to train the models into your models to get probabilities, which will lead to better probabilities than should be expected in real life and thus much better than should be expected results on your 2nd stage model (i.e.: overfitting). To remedy this you must do everything within a cross validation loop, that looks like this in pseudo code:

for <fold> in <crossvalidation>:
     <fold predictions> = random_forest.fit(<test_data_fold>)

Doing things this way will emulate real life since your predictions will be gathered using models that did not include that data when training them. You can build your final model using all of your prediction variables as input. If you struggle with actually executing on this either edit your question with code or post a new question on how to run everything with a CV loop.


try cross-validation on training data, cross-validation combines (averages) measures of fit (prediction error) to correct for the optimistic nature of training error and derive a more accurate estimate of model prediction performance (eg : k-fold), check accuracy (Precision/recall/f measure) and then use this classifier on test data.

  • $\begingroup$ if he's using a randomly selected test set there is no reason why using the same methodology within CV should cure a massive difference in results. $\endgroup$
    – David
    Aug 17, 2015 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.