Am I overfitting my random forest model (more information in description)?

Question

First off, sorry if this a novice question! Relatively new to all this stuff. Posted this in Stack Overflow and someone sent me here! Hope it's the right place.

Anyway, I'm working with 22 datasets that each have 180 observations of "Oddball" data and 720 observations of "Standard" data. I'm trying to use random forests for classification (i.e., oddball=1, standard=0). I understand there should be approximately equal trials/observations for both factors, but if I use 75% of the oddball data, then I'm barely using over 18% of the standard data. These data are pretty variable, and I think this could be problematic.

If I make four models, still using the same training data for each, am I overfitting my model? There's a lot more I've written, but this is basically what I'm trying to do:

jj = sample(1:180,(180*75),replace = F #Take 75% of all oddball data
kk = sample(181:900,(720*.75),replace = F) #Take 75% of all standard data
jj = sample(jj); kk = sample(kk) #Mix them up
kk = matrix(kk,4) #Divide the standard data so there are 4 sets of equal numbers for jj
  
samp1 = c(jj,kk[1,])
samp2 = c(jj,kk[2,])
samp3 = c(jj,kk[3,])
samp4 = c(jj,kk[4,])

I would then make four models (all while not touching the out-of-sample data) using each of these sample sets and average all their predictions to give me a "master" probability (i.e., an average of .8 would be deemed an oddball).

Is this overfitting the data? Is it even possible to overfit the data when using random forests? Is there something wrong with this intuition?

Thank you for anyone that helps! Your time and expertise is much appreciated.

Answer 1

I dont think its necessary combining 4 models into one by averaging probabilities (how do you know that they have same weights? let the learner handle those weights) since you are using same features and learner which is an ensemble (it combines weak learner into one natively already). Therefore it's better to think about labeling and class balances of those 4 models.

Is this overfitting the data? Is it even possible to overfit the data when using random forests? Is there something wrong with this intuition?

In order to say that you are overfitting or not, train and test errors should be available. Simply training error is very low compare to test error it implies overfitting.

Secondly, ensemble models are likely to overfit due to their greediness. And finally, you mentioned a bit class imbalance. So you should check metrics such as roc auc, recall and precision according to your case rather than accuracy.

Therefore my suggestion is that:

1. Apply hyper parameter tuning for all 4 models.
2. Report mean training & test errors with relevant performance metric (for imbalanced consider to use roc auc).
3. If there are huge differences between test and training errors, try to change your model/feature set etc since you are overfitting (eg: .99 for training, .70 for test). Otherwise pick the best estimator among 4 models and use this as final estimator.

Final comment: As complexity increases, models are more likely to overfit. (logistic regression is less likely to overfit compare to ensemble trees such as rf, gbm, xgboost). Thus always tune your parameters when using ensembles.

Hope it helps!

Answer 2

This is more like a comment than an answer but my newbie status doesn't allow me to comment yet.

You might be interested in reading this article: Random forest

4 samples appears to me being low a number of samples, I would recommend that you try using more samples and compare your results.

Besides, there are some good implementation of the algorithm out there, are you trying to re-implement it for academic purposes?