When comparing a GBM to a logistic regression for a binary classification, there a pros and cons to each. I'm interested in understanding the general tradeoff between the length of the data set (number of rows) vs the reliability of the fit out-of-sample.
Obviously the more data, the more reliable the predictions will be out of sample (all else being equal). The more rows of data, the more likely that GBM will be more predictive on a real data set. So i'm wondering is there a rough rule of thumb when you would say the data is too small to use GBM, better of using a logistic regression. I know that logistic regressions can be more predictive in some cases, but my understanding is that GBM usually outperforms.
I'm particularly interested in the case that the number of predictive variables is far less than the total number of variables available. For example, we have 100 variables, 10 of which might be highly predictive of the target variable.
I know about all the benefits of train/test sets and CV. But that doesn't work all the time either. For example, your colleague has just run the train/test loop too many times, has added too many unreliable variables into the mix to improve the test AUC, and now you have a GBM model that is crazy good. Too good.