Why does Bagging or Boosting algorithm give better accuracy than basic Algorithms in small datasets?

Question

I was working with a small dataset, with 392 values, and it was kind of an imbalanced dataset, with 262 values belonging to class 1 and rest 130 to class 0. So I did the upsampling technique, importing sklearn.resampling module.

However, the total dataset was now around 520 values. I applied basic, algorithms first like Logistic Regression and SVM Classifier, and since we all know that precision is not a good accuracy metric for imbalanced dataset, I use the f1-score and recall score. In logistic Regression I found out, it was giving 78% f1-score for class 1 and 80% for class 0 , and almost 99% f1-score for class 0 in SVM and 72% for class 1, which shows that it is overfitting.

But to my surprise I found out that Random Forest gave me a better accuracy, with having around 83% f1-score for class 0 and 82% for class 1 . But till now everywhere I have seen that for bagging and boosting algorithms to work well, we need a lot of data, which is not the case in this scenario.

I've searched google a lot, but unfortunately I haven't been able to get any specific answer, and I need to know the fundamentals, why does this happen?

Logistic Regression:

                precision    recall  f1-score   support

          0       0.80      0.80      0.80        91
          1       0.78      0.78      0.78        82

avg / total       0.79      0.79      0.79       173

[[73 18]
 [18 64]] (confusion matrix)

SVM with rbf-kernel:

                precision    recall  f1-score   support

          0       0.80      0.99      0.88        91 (kind of overfitting for class 0)
          1       0.98      0.72      0.83        82 

avg / total       0.89      0.86      0.86       173

[[90  1]
 [23 59]]

Random Forest Classifier:

                precision    recall  f1-score   support

          0       0.82      0.86      0.84        87
          1       0.85      0.81      0.83        86

avg / total       0.84      0.84      0.84       173

[[75 12]
 [16 70]]

Answer 1

SVMs finds a boundary to divide the data between the two classes, and the rule of thumb is to use SVMs for binary classification problems. Typically, when you have linear dependencies SVMs performs better than random forest since there is a distance to measure. But random forest performs better with categorical values / non-linear dependencies. Random forest does benefit from more data, but nature of data even in small data-sets matters.

Logistic regression follows similar logic, if your underlying relation is linear in nature then it probably perform better than Random Forest.