Comparing RMSEs of multiple test sets having different sizes

Question

The data I have is a time series data (stock returns), and I am training a Random Forest Regressor on it. Total observations = 2499

To better evaluate the performance, I have implemented rolling windows testing with training window sizes = 500, 700, 900,..., 2100. Though instinctively it would seem obvious to choose a window size which produced lowest RMSE, how can I be sure that the comparison is fair?

I mean with increasing window size, the test set size decreases. With window size 500, test set size is 1999. With window size 700, test set size is 1799.

I think the same question applies to Expanding Window

So is it sensible to compare RMSEs when test samples are decreasing in size?

If not, then how should one choose the best training window?

Answer 1

The "mean" in RMSE ensures that value of this metric are comparable and of the same scale irrespective of the size of the test window.

Have you looked at k-fold cross validation? That allows you to use a single train test split ratio of say 70:30, yet create k different datasets to compute the RMSE on.

K-fold cross validation is widely adopted and well researched. It's probably better known strategy than what you describe.