STRUCTURED DATASET -> MISLEADING OOB ERRORS
I've found interesting case of RF overfitting in my work practice. When data are structured RF overfits on OOB observations.
I try to predict electricity prices on electricity spot market for each single hour (each row of dataset contain price and system parameters (load, capacities etc.) for that single hour).
Electricity prices are created in batches (24 prices created on electricity market in one fixing in one moment of time).
So OOB obs for each tree are random subsets of set of hours, but if you predict next 24 hours you do it all at once (in first moment you obtain all system parameters, then you predict 24 prices, then there is an fixing which produces those prices), so its easier to make OOB predictions, then for the whole next day. OOB obs are not contained in 24-hour blocks, but dispersed uniformly, as there is an autocorrelation of prediction errors its easier to predict price for single hour which is missing then for whole block of missing hours.
easier to predict in case of error autocorrelation:
known, known, prediction, known, prediction - OBB case
known, known, known, prediction, prediction - real world prediction case
I hope it's interesting.