I have N models, each of them are used to predict on a set of data. I am currently combining their predictions by averaging across rows. Need suggestions on combining their predictions. End goal is to classify the samples.
Part of the problem lies in how much data you have. To create a second level of complexity, you ideally want to use a holdout data set to decide the right combination for the model predictions. If you use the training data from the models themselves to also combine the model output, you risk over fitting your final model. If you have a small data set, trying to build an ensemble on top of it might lead to worse performance in deployment since the model is only memorizing the training data. In that case, a simple average might do you better than trying to anything more sophisticated.
But, assuming you have more data to use, or if you want to use the data you trained on anyways, you can either design a weighted average or create a second model.
If you have an idea of which model performs better, you might want to try manually assigning weights for each model's output. A simple average assigns an equal weight to each output, but you can experiment with shifting the weight around a bit.
For an example, given two models, a basic average yields
new_pred = 0.5 * mod_1 + 0.5 * mod_2
However, if you see the performance on model one does better overall, you might shift the average to
new_pred = 0.9 mod_1 + 0.1 mod_2
If instead you want to create a second model, you can try using any machine learning model suitable for classification. I would probably stay away from larger models like Random Forests, but doing something like using the predictions as features into a Logistic Regression model can work well. But again, I would emphasize only trying to use the predictions from a holdout data set into the logistic regression model to make sure you aren't over fitting.