I am running an ML classifier on my data. I used SVM, RF and KNN. I used GScv for each of them and then used votingclassifier.The accuracy i got in each classifier independently was low, but from the hard and soft vote of the voting classifier is much higher! Why is that?

Here is my code

Fitting Kernel SVM to training set

from sklearn.svm import SVC
clf = SVC(C=1, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma=0.005, kernel='rbf',
  max_iter=-1, probability=True, random_state=1, shrinking=True,
  tol=0.001, verbose=False)

Checking accurancy, Best score of GV

print('Checking accurancy, Best score of GV')
best_accuracy =  grid_search.best_score_ 
print (best_accuracy)

[out] Checking accurancy, Best score of GV

Applying RandomForest (RF) Classification

from sklearn.ensemble import RandomForestClassifier

clf_rf = RandomForestClassifier(random_state=42)
clf_rf.fit(X_train, y_train)
clf_rf.score(X_test, y_test)
print ("Mean accuracy is", (clf_rf.score(X_test, y_test)))

y_pred2 = clf_rf.predict (X_test)

from sklearn.metrics import accuracy_score
accuracy_score(y_test, y_pred2)

[out] accuracy 0.21

Applying KNN

from sklearn.neighbors import KNeighborsClassifier

neigh = KNeighborsClassifier(n_neighbors=10)
neigh.fit(X_train, y_train) 

print (neigh.score(X_train, y_train))

[out] accuracy 0.58

Predicting the test set results

y_pred3 = neigh.predict (X_test)

from sklearn.ensemble import VotingClassifier

eclf1 = VotingClassifier (estimators= [('svm', clf), ('rf', clf_rf), ('KNN', neigh)], voting = 'hard')

eclf1 = eclf1.fit(X_train, y_train)

print 'Hard vote accuracy =', eclf1.score(X_train, y_train)

eclf2 = VotingClassifier (estimators= [('svm', clf), ('rf', clf_rf), ('KNN', neigh)], voting = 'soft')

eclf2 = eclf2.fit(X_train, y_train)

print 'Soft vote accuracy =', eclf2.score(X_train, y_train)

Hard vote accuracy = 0.9830508474576272
Soft vote accuracy = 0.7711864406779662

Basically I would say that your individual classifiers might be overfitting the training data. Ensemble voting is known to avoid overfitting, so it does not seem unlikely that the ensemble classifier behaves better than each classifier individually.

  • $\begingroup$ Could it be that the voting classifier is the overfit one and not doing a better job? $\endgroup$ Oct 17 '18 at 0:28
  • $\begingroup$ Matching several classifiers is expected to reduce the overfitting effect. In the worst case prediction, all classifiers are wrong, but the voting prediction is not worse than the individual ones. $\endgroup$ Oct 17 '18 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.