How to ensemble classifier incorporating all features in python?

Question

I am doing a text classification task(5000 essays evenly distributed by 10 labels). I explored LinearSVC and got an accuracy of 80%. Now I guess whether accuracy could be raised by using ensemble classifier with SVM as base estimator?

However, I do not know how to employ an ensemble classifier incorporating all the features? Please note that I do not want to combine the different features directly in a single vector.

Therefore, My first question: in order to improve the current accuracy, is it possible to use ensemble classifier with svm as base estimator? My second question How to employ an ensemble classifier incorporating all features?

Answer 1

Boosting - is the ensemble which tries to add new models that do well where previous models lack. Bagging in scikit lets you send the base classifier as the parameter. Go through 1.11 of the link to understand more!

But since, you already have in mind that SVM performs better Voting Classifier which is present in sklearn.ensemble lets you give weights to the classifiers which you seem to be performing well. For instance in your question, SVM can be given more weight. It also has another parameter 'voting'. If hard, uses predicted class labels for majority rule voting else if soft, predicts the class label based on the argmax of the sums of the predicted probabilities.

Answer 2

Yes, it is certainly possible to improve accuracy with SVM.

First, I'd go for already prepared ensemble algorithms like Random Forest or AdaBoost. They can incorporate all features and are strong predictors.

Further, as mentioned from Hima Varsha answer, scikit-learn offers voting_classifier to easily combine algorithms. But you can go a step further and do something what is called blending - You fit various algorithms on your data set and then, on their prediction, you fit another set of algorithms to produce final prediction. It can be easily implemented, see this example.