Ensemble model overfitting?

Question

I am attempting a classification project. I have split my data, 20000ish, into training and test sets. On the training sets I run a selection of classifieds including knn random forests and gbm . These give me about 20-30% accuracy at best. For each sample I generate probabities of each class and make a new model

Knn proba 1, knn proba 2.... Random forests proba 1 etc

On this I then run a random forest classifier which gives me a 90ish% accuracy against the test set.

Fantastic!... But when I use the model against new data the accuracy is very poor.

In part it feels like a case of overfitting but surely the test set should be poor as well

Why might the test data so good but new data so poor... What have I done wrong?

Thanks Chris

Answer 1

Yes I suspect you are overfitting. When you build your first stage of models (nearest neighbors, random forest, gradient boosting, etc...) is the process like this?

random_forest.fit(train_data,target)
random_forest_probabilities = random_forest.predict_proba(train_data)
...

If so you are plugging in the same data you used to train the models into your models to get probabilities, which will lead to better probabilities than should be expected in real life and thus much better than should be expected results on your 2nd stage model (i.e.: overfitting). To remedy this you must do everything within a cross validation loop, that looks like this in pseudo code:

for <fold> in <crossvalidation>:
     random_forest.fit(<train_data_fold>,<target_fold>)
     <fold predictions> = random_forest.fit(<test_data_fold>)
     ...

Doing things this way will emulate real life since your predictions will be gathered using models that did not include that data when training them. You can build your final model using all of your prediction variables as input. If you struggle with actually executing on this either edit your question with code or post a new question on how to run everything with a CV loop.

Answer 2

try cross-validation on training data, cross-validation combines (averages) measures of fit (prediction error) to correct for the optimistic nature of training error and derive a more accurate estimate of model prediction performance (eg : k-fold), check accuracy (Precision/recall/f measure) and then use this classifier on test data.