I have built several models on the training dataset and i am not happy with the results and I wish to club them all together and generate a new model, so here is my idea as i already have the results for the existing models i would like to create a new dataset with the existing model results as separate features on top of the original feature dataset applying a clustering to filter some data in the original dataset and would like to train the model across all the same models and get the result, Would that be called as stacked modelling?
2 Answers
$\begingroup$
$\endgroup$
9
Stacking takes predictions from diverse and shallow or weak models on a dataset.
Stacks meta-features ( meta-features = predictions ) like columns. And usually a linear meta-model ( like Linear Regression ) is fitted on that dataset of metafeatures. Think of it as if you let multiple models , each one with his own prediction , decide what's the best value for each datapoint. mean across all models? maybe mean across just two? The meta-model decides.
Your approach of using meta-features with the original features ressembles to Boosting which takes each datapoint's residual ( difference between truth value and prediction ) and uses it as a feature to correct the gap iteration by iteration.
-
$\begingroup$ I am still confused, I use the same training dataset and features for training on various classifier algorithms and combine the results and use another classifier model to get the predictions, is that what you understood by the question? Or did I convey it wrong? $\endgroup$– DevOct 21, 2019 at 7:24
-
$\begingroup$ when you use the last classifier , do you use the original features again or only the predictions you got from the level 1 classifiers? $\endgroup$– BlenzOct 21, 2019 at 8:33
-
$\begingroup$ I use the original features along with other features as well. $\endgroup$– DevOct 21, 2019 at 9:28
-
$\begingroup$ Well then it's not stacking. It's similar to boosting but you're performing only one iteration. If you want to apply a stacking scheme, get predictions from diverse first level models on your test set and your whole train set using a portion of your train set, use the other portion to train a second level model using as features only the predictions you got from your first level models ( 2nd level should be a simple model like linear regression ) and predict for test . $\endgroup$– BlenzOct 21, 2019 at 9:35
-
$\begingroup$ The others feature mentioned earlier are basically the classifier results where each classifier is a feature along with the initial features. $\endgroup$– DevOct 21, 2019 at 9:43
$\begingroup$
$\endgroup$
4
Would that be called as stacked modelling?
Yes, this is exactly what stacking models means.
I didn't understand what is the role of clustering in your design?
The standard approach consists in using only the predictions of the $N$ individual learners as feature for the meta-model. The training/testing data split is a bit more complex: the training of the meta-model requires predictions from the individual models, so the data could be split like this:
- training data for individual models
- training data for the meta-model (on which individual models are applied)
- testing data for the full system
Generally it's safer to choose a simple learning algorithm for the meta-model, because there is a higher risk of overfitting (for instance linear regression or majority voting).
[Detailed answer to comments below]
The standard setting would be like this:
- Train $N$ individual models $m_1,...,m_N$ using first-level training set $T_1$
- Apply these models in order to obtain $N$ features for the meta-classifier. At this stage it's a bad idea to use instances from $T_1$, because the individual classifiers have been trained on $T_1$. so one needs a second set of instances $T_2$: for every instance $x\in T_2$, let $m_1(x),...,m_N(x)$ be the predictions resulting from applying classifiers $m_1,...,m_N$.
- Train the meta-classifier model $M$ using $T_2$ as training set, with $m_1(x),...,m_N(x)$ the features for any $x\in T_2$.
- Testing: as usual one needs a fresh set of instances, say $T_3$. For any instance $x\in T_3$ calculate the prediction by applying $M$ to the $N$ predictions obtained by the individual models, i.e. $M(m_1(x),...,m_N(x))$
-
$\begingroup$ Can i use the complete/full training data/features for generating all meta-models ? will that still be a stacking model ? $\endgroup$– DevOct 18, 2019 at 14:24
-
$\begingroup$ You can but there's a big risk of overfitting (or some kind of similar issue, I'm not sure if it's called overfitting in this case): that's because the individual models have seen the data during training so they can give very good predictions for these instances; now when the meta-model is trained it is provided with these good-quality predictions, so that's what it expects and what it is trained to deal with. But when testing the individual models are going to make more/different errors, and the meta-model won't be able to deal with these. $\endgroup$– ErwanOct 18, 2019 at 14:37
-
$\begingroup$ I am still confused, I use the same training dataset and features for training on various classifier algorithms and combine the results and use another classifier model to get the predictions, is that what you understood by the question? Or did I convey it wrong? $\endgroup$– DevOct 21, 2019 at 7:26
-
$\begingroup$ @Dev I'm not sure myself if we are talking about the same thing because your original question includes some clustering step that I don't understand. I edited my answer to show you what would be the standard stacking approach (as far as I know), so that you can see if we are talking about the same thing or not.. let me know $\endgroup$– ErwanOct 21, 2019 at 13:07