We're using a whole year's data to predict a certain target variable.The model works like data - OneHot encoding the categorical variables - MinMaxScaler - PCA (to choose a subset of 2000 components out of the 15k) - MLPRegressor. When we're doing a ShuffleSplit cross-validation and everything is hunky-dory (r^2 scores above 0.9 and low error rates), however in real life, they're not going to use the data in the same format (e.g. a whole year's data), but rather a subset that is due that month (e.g. in March they'll need the algorithm to predict rows that are due in March) and at this point, the algorithm fails.
The expectation is that we have some data (for example, 2018 December - 2019 December) that we're training on, and then we'll need to predict items due in January 2020 (that's not yet available). Again, if we do a random train-test-val split, the r^2 scores are great, but the team lead is not okay with this, as he said the real-life use case is that we train on 2019's data and then predict on a monthly basis, so he wants us to train on not all the data, but only 2018 December to 2019 November, and then validate on 2019 December data.
I see how this is methodically incorrect, as we're training on data that has different properties (whole year's data vs a certain month's due data), and this reflects on the validation scores. If I isolate a certain month to simulate the real life scenario (e.g. train on 11 months of data and test on one specific month), depending on the month we're getting r^scores between 0.3 and 0.7 depending on the month isolated (which is a far cry from the 0.9 if randomly sampled from the whole dataset).
What I can't figure out is, how can we structure this modelling so that the test data (items due in a specific month) has the same properties than the training set (the rest of the months, or maybe 12 months including 1 month from last year), while preserving information for the whole year? Or should we be doing 12 models for each month?
