How to average classifiers with AUC metric?

Question

I am modeling a binary classification and my loss function is the gini function (normalized area under the curve). Here's my implementation:

1. Split the data with k-folds
2. Train k classifiers

Now I have k classifiers, but I need one classifier. So the naive approach is:

$$prediction_i = \frac{prediction_{i1} + prediction_{i2} + ... + prediction_{ik} }{k}$$

There may be possible problems with this combining technique. For example, gini is scale invariant. I could take prediction1 and scale it with exp(p_i) and then scale prediction2 with sqrt(p_i). These would have zero affect on scoring the individually but it would mess up my combining step.

What is the most appropriate combining function?

Answer 1

Answer

In ensemble methods, the predictions are typically made by majority voting. Using python and Numpy.

import numpy as np

pred1= [.5, .5 , .0]
pred2= [.3, .2 , .5]
pred3= [.5, .1 , .4]

pred1 = (pred1 == np.max(pred1)).astype(np.int)
pred2 = (pred2 == np.max(pred2)).astype(np.int)
pred3 = (pred3 == np.max(pred3)).astype(np.int)
votes = pred1 + pred2 + pred3 # == array([2, 1, 1])
pred = np.argmax(votes) # pred == 0

Side Note

Also, K-Fold is generally used for hyper-parameter tuning. After you made a decision on the hyper-parameter you would refit the model on the entire dataset. If you want to use ensemble methods like random forest or gradient boosted tree, you would apply the algorithm to the entire dataset.

It is not clear to me whether your method of ensembling will have any benefit over the standard approach. In your approach, each classifier is trained on the smaller set of data on all features available, and I can't think of any reason for us to do this.