Structuring experiment/training data with months in mind

We're using a whole year's data to predict a certain target variable.The model works like data - OneHot encoding the categorical variables - MinMaxScaler - PCA (to choose a subset of 2000 components out of the 15k) - MLPRegressor. When we're doing a ShuffleSplit cross-validation and everything is hunky-dory (r^2 scores above 0.9 and low error rates), however in real life, they're not going to use the data in the same format (e.g. a whole year's data), but rather a subset that is due that month (e.g. in March they'll need the algorithm to predict rows that are due in March) and at this point, the algorithm fails.

The expectation is that we have some data (for example, 2018 December - 2019 December) that we're training on, and then we'll need to predict items due in January 2020 (that's not yet available). Again, if we do a random train-test-val split, the r^2 scores are great, but the team lead is not okay with this, as he said the real-life use case is that we train on 2019's data and then predict on a monthly basis, so he wants us to train on not all the data, but only 2018 December to 2019 November, and then validate on 2019 December data.

I see how this is methodically incorrect, as we're training on data that has different properties (whole year's data vs a certain month's due data), and this reflects on the validation scores. If I isolate a certain month to simulate the real life scenario (e.g. train on 11 months of data and test on one specific month), depending on the month we're getting r^scores between 0.3 and 0.7 depending on the month isolated (which is a far cry from the 0.9 if randomly sampled from the whole dataset).

What I can't figure out is, how can we structure this modelling so that the test data (items due in a specific month) has the same properties than the training set (the rest of the months, or maybe 12 months including 1 month from last year), while preserving information for the whole year? Or should we be doing 12 models for each month?

• Hi, did you think about using time series analysis () considering as well seasonal effects?smth like SARIMA? just an idea.. – zina Feb 12 '20 at 10:05
• Not yet, might look into it, problem is that the data is not just impacted by month, but also our colleagues' strategy, and time series will not capture that. I'm thinking I'd somehow do a kernel density matching? But no idea how to transfer the density once I model it. – lte__ Feb 12 '20 at 10:23
• can you detect features influencing results and make them dependent on months? than you can have influence of previous months + impact of a month you are going to predict – zina Feb 12 '20 at 12:33
• Yeah but it's not just dependent on month... And by the time I do PCA etc, I end up with 750 features, and then it's still influenced by other variables... I just want to try this, I do have alternative approaches in mind. I do want to focus on what I'm asking in this question in this thread. – lte__ Feb 12 '20 at 12:40