4
$\begingroup$

I'm reading the following kaggle post for learning how to incorporate model stacking

http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/ in ML models. The structure behind constructing the 5 folds and creating out of sample predictions on the training data makes sense for the purpose of building the meta model or the model on top of the base models. However i'm not sure how it uses hyper parameter tuning especially for the base models.

So the concept of getting out of sample predictions makes sense to me. We essentially for each of the 5 folds use the other 4 folds to train and then predict on the fifth. So how do we actually hyper parameter tune the base models on this same dataset without adding bias, it's seems to me that this is not possible?

Note i'm making the assumption that there is no more data available to use. I'd appreciate any help!

Improve this question
$\endgroup$
1
  • $\begingroup$ I guess a usual strategy is to try different extreme versions of the hyperparameters, e.g. one model with a very large value in one parameters and very small value in anotoher, then the same model but with the opposite type of parameters. Tuning hyperparameters conjointly would be a reeeeaaally slow process and I've never heard of that being done, but you can still try optimization-based approaches such as gaussian process or bayesian optimization that would work with the parameters of all models simultaneously. A simple grid search would likely be intractable. $\endgroup$
    – anymous.asker
    Nov 17, 2018 at 18:43

1 Answer 1

Reset to default
1
$\begingroup$

Do not mix dividing the data into k-fold with cross validation.

You can use the 4 folds ( training data) to optimize the base classifiers. You can also find the best hyper parameters by applying cross validation on your training data, re-train using all training data ( the 4 folds) and then test using the last fold to generate the meta data.

After you finish generating the meta data you can now use all the data the 5 folds to train the base learners. The higher the convergence between the final set of base classifiers and the ones you used to generate the meta data the better. This is why the less the data you have the larger the k you need to go with and vice versa.

Finally, you use the meta data to train the meta classifier

Here is one last note to think about: using cross validation to optimize base learners my not be very beneficial, but why? If you have access to N different training algorithms and you can use cross validation to optimize them and choose the best one ( the one that leads to a very low bias error), then there is a high probability that you don't need to use stacking. But stacking is very beneficial when all these N algorithms can not lead to 0 bias error (if you have variance error use bagging), it is a case that may confuse the cross validation algorithm. The other thing is using cross validation to optimize algorithms may lead to different algorithms that have similar biases, which reduces the benefits from the stacking technique.

you need to know that stacking, is one of the most tricky ensembles and this is why it is not well studied in literature in compare to bagging and boosting. In the other hand it is proven that it is very important, especially for practical purposes.

Improve this answer
$\endgroup$
1
  • $\begingroup$ how would i be able to hyper parameter tune on my training data using C.V and then use that same data to build predictions for the meta model this makes no sense to me. For example suppose i have 3 models each with two features to hyperparameter tune. If i hyperparameter tune on the training data using cv then surely its biased to then use the best two models again on that same data to get predictions even if they are out of sample? This is because best model was selected on the training data itself? $\endgroup$
    – Iltl
    Nov 23, 2018 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.