# Formula to calculate confidence value in Adaboost

I am coding an AdaBoostClassifier with the two class variant of SAMME algorithm. Here is the code.

def I(flag):
return 1 if flag else 0

def sign(x):
return abs(x)/x if x!=0 else 1


class AdaBoost:

def __init__(self,n_estimators=50):
self.n_estimators = n_estimators
self.models = [None]*n_estimators

def fit(self,X,y):

X = np.float64(X)
N = len(y)
w = np.array([1/N for i in range(N)])

for m in range(self.n_estimators):

Gm = DecisionTreeClassifier(max_depth=1)\
.fit(X,y,sample_weight=w).predict

errM = sum([w[i]*I(y[i]!=Gm(X[i].reshape(1,-1))) \
for i in range(N)])/sum(w)

'''Confidence Value'''
#BetaM = (1/2)*(np.log((1-errM)/errM))
BetaM = np.log((1-errM)/errM)

w = [w[i]*np.exp(BetaM*I(y[i]!=Gm(X[i].reshape(1,-1))))\
for i in range(N)]

self.models[m] = (BetaM,Gm)

def predict(self,X):

y = 0
for m in range(self.n_estimators):
BetaM,Gm = self.models[m]
y += BetaM*Gm(X)
signA = np.vectorize(sign)
y = np.where(signA(y)==-1,-1,1)
return y


The much I know the formula for confidence is

The much I read, the actual minima occurs when c=1/2 but for any value of c the classifier should produce the same result. But when I am coding the class the output for c = 1 and c = (1/2) are coming different. Moreover if I am not multiplying anything ie. c=1 then the output of my classifier is better and produces identical results with the sklearn implementation of AdaBoost Classifier.

So why multiplying 1/2 is giving bad results?

Actually this equation is not quite right.

The actual minima occurs at

In the derivation this BetaM is the solution to the equation

Notice how in the function we are scaling the weights of wrong predictions up by exp(BetaM) and scaling down the weights of correct predictions by multiplying exp(-BetaM).

But in the code that I have written (also in the sklearn implementation) only the wrong predictions are being scaled up. So to get the relative scaling correct we have to scale the wrong predictions by exp(2BetaM) i.e.

And as of the hypothesis function has the form

Any scalar multiple of BetaM will work fine as all the predictions will be scaled by the same amount. So for convenience in coding BetaM is written as just

Note: You can perfectly use BetaM = 0.5 * log((1/errM)/errM) . But then you have to scale down the correct predictions too. If you do that then the code will give correct results.