AdaBoost is known to be sensitive to outliers & noise. However, the explanation seems to be hard to found or nontrivial.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Consider a supervised problem with target values y either being -1 or 1. Let's say we are fitting function f(x) using our AdaBoost classifier. Intuitively we want positive margins, where a margin is defined as $y_{i}f(x_{i})$. A positive margin means we have correctly classified that sample since the sign of both f and y are the same. A negative margin means we have misclassified the sample.
The loss function for Adaboost is defined as an exponential:
$L(y, f(x)) = exp(-yf(x))$
This exponential then ends up more strongly penalizing points with a large negative margin (i.e., outliers) than other classifiers. Here is a plot of the loss function vs. the margin for several classifiers:
So, to summarize: Adaboost is susceptible to overfitting outliers due to the form of its loss function.
Source: Hastie, et. al, The Elements of Statistical Learning, 2017, pp. 343-347
