2
$\begingroup$

I am working on kidney cancer patients' data with 5 unbalanced labels. These codes are contained of Normalization, Oversampling on Feature Engineering part. A list of 9 ordinary Machine Learning methods is provided which are used for the classification task. Then, I take advantage of two kinds of ensemble methods of hard voting and weighted voting methods. 10-fold CV has is exploited to validate results.

methods = ['Support Vector Machine', 'Logistic Regression', 'K Neighbors Classifier', 'Random Forest', \
          'Gaussian Naive Bayes', 'Linear Discriminant Analysis', 'Decision Tree', 'Gradient Boosting',\
          'soft_VotingClassifier','hard_VotingClassifier']

I would like to know how to tune weights for the soft voting method one? Here, are the code and results I have right now:

from sklearn import preprocessing
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.ensemble import VotingClassifier

if method == 'soft_VotingClassifier':
    cl1 = LogisticRegression()
    cl2 = KNeighborsClassifier(n_neighbors=10)
    cl3 = RandomForestClassifier(max_depth=35)
    cl4 = GaussianNB()
    cl5 = LinearDiscriminantAnalysis()
    cl6 = DecisionTreeClassifier()
    cl7 = SVC(C=0.1, gamma=0.0001, kernel='poly')
    cl8 = GradientBoostingClassifier()
    estimator = [(method[0],cl1), (method[1],cl2), (method[2],cl3), (method[3],cl4),\
                 (method[4],cl5), (method[5],cl6), (method[6],cl7), (method[7],cl8)]
    eclf = VotingClassifier(estimators=estimator,
    voting='soft', weights=[5, 5, 10, 5, 6, 8, 4, 10])

if method == 'hard_VotingClassifier':
    cl1 = LogisticRegression()
    cl2 = KNeighborsClassifier(n_neighbors=10)
    cl3 = RandomForestClassifier(max_depth=35)
    cl4 = GaussianNB()
    cl5 = LinearDiscriminantAnalysis()
    cl6 = DecisionTreeClassifier()
    cl7 = SVC(kernel='linear',gamma='scale')
    cl8 = GradientBoostingClassifier()
    estimator = [(method[0],cl1), (method[1],cl2), (method[2],cl3), (method[3],cl4),\
                 (method[4],cl5), (method[5],cl6), (method[6],cl7), (method[7],cl8)]
    eclf = VotingClassifier(estimators=estimator, voting='hard') 

Confusion Matrix results on test data:

[50  5  1  4  2]
[ 0 13  1  0  3]
[ 0  1  2  1  1]
[ 4  0  0  2  0]
[ 0  3  0  0  2]

Accuracy result on test data:

Accuracy result

For those kind people who want to check my code or even run them, I gonna put the repository link.

$\endgroup$
3
  • 1
    $\begingroup$ There is a similar question, you could possibly find an answer in the question itself. If you don't, let me know datascience.stackexchange.com/questions/85644/… $\endgroup$ Nov 24 '20 at 8:26
  • $\begingroup$ May I give you more tips? What kind of score should I define for 5 unbalanced classes? While I have used accuracy, I know this metric does not work well for 5 classes. When they are unbalanced can be even worse. $\endgroup$
    – amin
    Nov 24 '20 at 18:57
  • $\begingroup$ Dear @CarlosMougan, May you kindly, share your idea with me? $\endgroup$
    – amin
    Nov 30 '20 at 1:41
1
$\begingroup$

The question is a classical optimization problem.

To make it more simple, lets say that you have 2 models (m1 and m2). And you want to do an ensemble.

First idea, is doing the mean of both

(m1.predict(X) + m2.predict(X))/2

Then you want to optimize the best combination of both models.

for w in [0,0.25,0.5,0.75,1]:
     (w * m1.predict(X) + (1-w) * predict(X) ) / 2 
     print(evaluate results/MSE/Accuracy...)

Your problem is exactly the same as this one but using the voting classifier.

You just need to optimize the array. In a for loop.

weights=[5, 5, 10, 5, 6, 8, 4, 10]

In this case, your array is not normalized and it has more models. So it will take longer. Also voting classifier fits every time the model, so I will take you a long time. Is easier to code it yourself.

$\endgroup$
2
  • $\begingroup$ Dear @CarlosMougan, As I understand from your answer, I need to normalize my array for the first step. I mean their summation be always equal to 1 (a + b + ... + h + I = 1). Then try to optimize this array of weights. But, the remind question is how to optimize this array in a loop? Do you have an idea to write an optimizer for doing so? $\endgroup$
    – amin
    Dec 1 '20 at 17:40
  • 1
    $\begingroup$ You can just implement it yourself. The most basic way is a for loop. $\endgroup$ Dec 1 '20 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.