1
$\begingroup$

I was wondering about whether it is possible to train a ML model for a classification task with dataset D, and then train another model to solve the same classification task, which takes as input dataset D and the output of the first model. I am worried about the data partition in this case (i.e. whether the models need to be trained and tested with different data samples). If this kind of ensemble method exists, how is it called?

As further details, I have a KNN model and a random forest to predict the same label, but they are trained with different variables.

$\endgroup$

2 Answers 2

2
$\begingroup$

Do you mean using an ML model to make predictions, and then combining the original features + predictions as new features into another ML model?

Conceptually that's a bit weird, unless you think the first ML model has some limitations (e.g. systematically biased) and you want to somehow correct for it using the second one. But in that case why not just use the second architecture.

A much more common type of ensembling is simply training both models, and then do some sort of aggregation (e.g. average) on the predictions.

$\endgroup$
2
  • $\begingroup$ Yes, that's what I meant! I will consider the voting ensemble you are suggesting. The main model is a random forest. Moreover, I have the coordinates of each record, and it seems that the label of the neighbourhood is useful to estimated the label of that particular record; I captured this idea with KNN. That's why I wanted to combine the power of both models. $\endgroup$ Apr 18, 2022 at 10:35
  • $\begingroup$ Ah that makes sense, especially with KNN as the first model. Just be careful with overfitting! $\endgroup$
    – Adam
    Apr 20, 2022 at 5:58
1
$\begingroup$

Stacking comes closest to what you describe. From the sklearn docs:

Stacked generalization consists in stacking the output of individual estimator and use a classifier to compute the final prediction. Stacking allows to use the strength of each individual estimator by using their output as input of a final estimator.

Stacking is one type of "ensamble learning" where you aim at combining several models into one final classifier. The idea is to take advantage of the ability of different models to capture different details in the data.

$\endgroup$
1
  • $\begingroup$ Great! I have just read a little bit how stacking is is implemented in sklearn and it is possible to train the base models with the same dataset as the meta-model, but concatenating as new columns the outputs of the base models (by using "passthrough" parameter). $\endgroup$ Apr 18, 2022 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.