I have N models, each of them are used to predict on a set of data. I am currently combining their predictions by averaging across rows. Need suggestions on combining their predictions. End goal is to classify the samples.
$\begingroup$
$\endgroup$
1
asked Feb 6, 2018 at 12:18
-
$\begingroup$ Do have a look at these threads : stackoverflow.com/questions/46074668/… stats.stackexchange.com/questions/255230/… I think it would really help if you could elaborate on what is the problem you are solving and the machine learning technique you are using to solve, in order to answer the question. $\endgroup$– Nischal HpCommented Feb 6, 2018 at 13:58
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Part of the problem lies in how much data you have. To create a second level of complexity, you ideally want to use a holdout data set to decide the right combination for the model predictions. If you use the training data from the models themselves to also combine the model output, you risk over fitting your final model. If you have a small data set, trying to build an ensemble on top of it might lead to worse performance in deployment since the model is only memorizing the training data. In that case, a simple average might do you better than trying to anything more sophisticated.
But, assuming you have more data to use, or if you want to use the data you trained on anyways, you can either design a weighted average or create a second model.
If you have an idea of which model performs better, you might want to try manually assigning weights for each model's output. A simple average assigns an equal weight to each output, but you can experiment with shifting the weight around a bit.
For an example, given two models, a basic average yields
new_pred = 0.5 * mod_1 + 0.5 * mod_2
However, if you see the performance on model one does better overall, you might shift the average to
new_pred = 0.9 mod_1 + 0.1 mod_2
If instead you want to create a second model, you can try using any machine learning model suitable for classification. I would probably stay away from larger models like Random Forests, but doing something like using the predictions as features into a Logistic Regression model can work well. But again, I would emphasize only trying to use the predictions from a holdout data set into the logistic regression model to make sure you aren't over fitting.
$\begingroup$
$\endgroup$
Is it a classification problem or a regression problem?
If you use average you assume equal weight for each model. You can use cross validation to identify what weights work best for you.