Understanding of stacking

Question

I use various autoML solutions so I can't stack my models directly (for example via StackingClassifier, mlextend or as layers in keras) so I want to implement my own pipeline for this case using only models output files.

The problem is that I have absolutely no idea how this stacking works. Don't misunderstand, I've read a few guides from the links, but still don't understand what to do with the data below.

We have, let's say, 3 models: DNN, XGB and RF, as well as the input data. Since the metafeatures already contain information about the training of basic models, we cannot simply divide the data by test / train, because there will be overfitting. Accordingly, we must allocate some of the data to meta_test to avoid this. Now we start learning. There are three models, so we have to split base_data into three folds, study the model on two, test on the last. So we do 3 times, like in cross-validation.

So far the breakdown looks like this:

After learning we want to get meta-features, so we make predictions using meta-data. Than we need to feed meta-features to meta-model:

And there is main question: HOW? If we imagine that meta-f are our train, what is test? We've spent all of our data before. Could someone tell me which stage in stacking I understand fundamentally wrong? Thanks for any help.

Links of used guides: https://scikit-learn.org/stable/modules/ensemble.html#stacking, https://alexanderdyakonov.wordpress.com/2017/03/10/c%D1%82%D0%B5%D0%BA%D0%B8%D0%BD%D0%B3-stacking-%D0%B8-%D0%B1%D0%BB%D0%B5%D0%BD%D0%B4%D0%B8%D0%BD%D0%B3-blending/

Answer 1

In short brief, stacking involves training multiple base models on different subsets of the training data, obtaining their predictions (meta-features), combining these predictions with the true labels to create meta-data, and training a meta-model on this meta-data. The meta-model then makes predictions on unseen data (holdout set) to assess its performance.

Note: It's important to avoid data leakage and ensure that the base models are trained on different subsets of the training data during the cross-validation process.

I hope this explanation helps clarify the steps involved in the stacking process :)

so let's break it down step by step to clarify the process.

1. Splitting the Data:

   • Initially, you have your input data that consists of features and corresponding labels.
   • You need to divide this data into two parts: the training set (base_data) and the holdout set (meta_test). The holdout set will be used to evaluate the performance of the stacked model.

2. Training Base Models:

   • Choose your base models, such as DNN, XGB, and RF.
   • Implement a cross-validation strategy (e.g., 3-fold cross-validation) on the base_data.
   • For each fold, train the base models on two parts of the base_data and make predictions on the remaining part (test fold).
   • Collect these predictions (meta-features) for each base model.

3. Creating the Meta-Data:

   • Combine the meta-features obtained from the base models with the corresponding labels from the test folds.
   • This combined dataset will be your new meta-data, where the meta-features are the input features, and the labels are the target variable.

4. Training the Meta-Model:

   • Choose a meta-model, such as logistic regression, random forest, or gradient boosting.
   • Use the meta-data (meta-features and labels) to train the meta-model.
   • The meta-model learns to predict the target variable based on the meta-features provided by the base models.

5. Evaluation on the Holdout Set:

   • Once the meta-model is trained, use it to make predictions on the meta_test set (holdout set) that was initially separated.
   • Compare the predictions with the true labels in the holdout set to evaluate the performance of the stacked model.
   • You can use appropriate evaluation metrics, such as accuracy, F1 score, or any other suitable metric for your problem.

---

some questions about stacking:

• Should we use the same fold number as number of models we use?
• And if we use base_test fold for meta_features, doesn't that mean that meta_features will have "encoded" data from base_train?
• Shouldn't we create specific fold, that no model will use in test_train process, and use it only to get meta-features?

Answer:
In the "Training Base Models" step of the stacking process, you are correct that the data is divided into folds for cross-validation. The number of folds you choose does not necessarily have to be the same as the number of models you use, although it is a common practice to have a one-to-one correspondence between folds and models.

The purpose of cross-validation is to assess the performance of your models and prevent overfitting. Each fold represents a subset of the training data that will be used for training the models, while the remaining part (test fold) is used for evaluating the model's performance.

During each fold of the cross-validation process, you train the base models on two parts of the base_data and make predictions on the test fold. The predictions from each model on the test fold are considered as the meta-features for that particular fold. These meta-features capture the information learned by each base model on the specific subset of the training data it was trained on.

Now, regarding your question about using base_test fold for meta-features, you're correct that this would result in meta-features containing "encoded" data from the base_train. This is not desirable because it can lead to data leakage, where the meta-model may inadvertently learn to exploit patterns present in the base_train data that are not generalizable to unseen data.

To avoid data leakage and ensure that the meta-features reflect the models' predictions on unseen data, it is recommended to create a specific fold that is not used in the training process of the base models. This fold is typically called the meta_train set. The base models are trained on the remaining folds (excluding the meta_train set) and their predictions on the meta_train set are used as meta-features. These meta-features are then combined with the corresponding labels from the meta_train set to create the meta-data.

keep in mind it is good practice to create a separate fold (meta_train set) that is not used during the training of base models but is used exclusively for obtaining meta-features. This helps to ensure that the meta-features represent the models' predictions on unseen data, allowing the meta-model to learn general patterns and make accurate predictions on new, unseen data.

Hope it will be useful for you.