# 2 stage ensemble -- CV MSE valid in 1st stage but not in 2nd

I'm trying out a Kaggle competition, which puts me in the unusual position of being able to get feedback on my models' "true" performance (you can submit several predictions per day and they give you your score on a performance metric -- in this case MSE -- each time).

As such, I've been observing a situation I can't explain -- my cross-validation results for my first stage Random Forest model, constructed with sklearn in Python, are consistently accurate within 1 percentage point of the true MSE.

For example, my first stage model will give a CV MSE of about .22 and the "ground truth" (actual) MSE will turn out to be about .23.

OTOH, my 2nd stage Random Forest, which is a different type of RF constructed in R, gives way "better" (smaller) training and cross validation MSE errors, like 0.05, but consistently turns out to have a much higher true MSE, like .23. In the 2nd stage Random Forest I have the predictions from the first stage and a few additional engineered features as regressors.

Is there a statistical explanation for this?

For example is it the result of some sort of recursion from stacking the models? If so, given that most winners of these Machine Learning contests are stacking many models, what's the appropriate way to do so that would avoid this statistical problem?

To clarify: you build one random forest on training data and get some results which seems to have no overfit, since CV and test results are similar. The second RF is built on the predictions of the first RF and some other additional features. The error on CV training data is very low and on test data are very high.

Now we will analyze the performance of the second RF without considering how it was built. Usually if you fit data very well on training set and very poor on test set it means you simply overfit data. This happens when your model learns too much from the data. The point is that in your training data you have two kinds of information. The first one is the real structure of the data, and the second one is the noise around that data, assuming that you have enough data and your model is able to handle the useful structure. The ideal case happens when you learn only the useful structure and when leave out the noise.When that happens you will have close errors in training and test data, which means that what your model learnt is at least almost only useful data. When you learn too much, which means you learn useful and noise together, your training error decreases because the results have the noise structure which is in training data, and on test data the errors grows since the noise from test data does not look like the noise from training data.

Now the first learner looks like it does not over fit. This of course does not always mean that you can't do better. It only means that what your learn is a structure which is also found in test data and thus has predictive power. So is understandable that you aim to improve the performance of the first predictor. One way would be to try to improve that predictor, and you can search for that by studying how that model learns. For example you can study if that model improves performance with more data by training repeatedly on samples of increased size and see if stabilize itself or not. There are many approaches. Another approach would be ensembles and a prominent one is stacking.

But you did not understood the idea of stacking. The whole point of stacking can be expressed in the following statement: explore the model space by building various different model families and in the end get the best from all of them. So stacking starts with the assumption that since we do not know which would be an ideal model for our data, we should try many imperfect ones. Some of them will behave well in some regions of space, some others in other regions of space, but none of them will be able to handle all the space at high performance. The key point is to have models from different families. Sometimes the same algorithm can produce very different models, and sometimes not. For example two SVMs with very different kernels can produce totally different surfaces. Sometimes even for the same kernel an SVM can produce very different results for very different complexity parameter. Often for RF this is not the case, but it is not impossible to get different surfaces with different parameters on RF. So the conclusion is that stacking aims to explore different ways of learning.

What you have done is to apply stacking on a single base learner by adding new features. First of all is that in general it does not work. The predictions from the first learner has a string structure, it comes from averaging on regions. So, this predictions compared with additional variables from the second model are more powerful. This means that every wrong thing learned in the first RF becomes very powerful on the second RF, because it uses the same averaging over regions procedure for learning. So the errors from the first learner amplifies in the second learner. I do not know about the additional features, but in this conditions it looks like it is very easy to overfit.

Turning back to what's in a data set: this data set has a structured content which you have to learn and an unstructured content (noise) around that. Your purpose of learning is to learn only the structure. The first RF is able to distinguish the structure from noise simply because the noise is not altered. The second RF is not able anymore to make that distinction, since the predictions of the first RF are already smooth in regions, which means it has a powerful structure, at least from the point of view of the second RF, which learns in the same way.

A last idea is that it looks like what you have done is somehow similar with a boosting procedure. The first learner gives you a structure which is used as a baseline by the second learner. But there is a big difference than in boosting. While in boosting you have a base line which you aim to improve, you improve that baseline by looking again at the original features, which have the noise unstructured. In your case at the second iteration you do not have again that chance and everything is structured. Thus errors amplifies.

My 2 cents: try multiple learners, eventually using additional features in those learners, try different models for base learners, start with very simple (low complexity or small degrees of freedom for) the stacking learner, like a penalized logistic regression and more further to more complex ones in a controlled fashion.

• Many thanks, Rapaio. I tried this with different types of models, however I had the same problem with MSE. RF seems to work best for this particular use case. The reasons that I'm doing stacking are 2-fold -- one is that there's features which I can only create in R or Python and the is that while these learners are both in the RF family, they are very different implementations and have different parameters. I can afford to run a much bigger forest in R than in Python, for instance. Thus they are different learners, just in the same general family of models. Feb 19, 2016 at 15:28