I am attempting a classification project. I have split my data, 20000ish, into training and test sets. On the training sets I run a selection of classifieds including knn random forests and gbm . These give me about 20-30% accuracy at best. For each sample I generate probabities of each class and make a new model
Knn proba 1, knn proba 2.... Random forests proba 1 etc
On this I then run a random forest classifier which gives me a 90ish% accuracy against the test set.
Fantastic!... But when I use the model against new data the accuracy is very poor.
In part it feels like a case of overfitting but surely the test set should be poor as well
Why might the test data so good but new data so poor... What have I done wrong?