How to deal with temporal trend in ML

Question

I am fitting a binary classifier and I observe a temporal trend in the response variable, meaning that the actual percentage of positives fluctuates with time, I can see periods where it is high and periods where it is low. Note that most of my variables are not time related and that this fluctuation does not seem to be due to a change of mix in other variables. And this trends does not seem to be seasonal or cyclical either, so I don't want to use a Time Series model. I would like my model to know this so that its average prediction on new data points is more or less the average percentage of positives observed on recent points. I thought of doing this in several ways, but none of them are satisfactory:

• Train my model only on recent data (e.g. less than 6 month old) --> but then I am possibly missing a lot of information only contained in old data
• Assign a bigger importance to newer observations, by artificially including more of recent observations or by using the sample_weights setting of ML libraries (available on sklearn and XGboost) --> But it has basically the same disadvantage as the previous method. Old information is potentially lost
• Create a new variable "time_since_start" that grows indefinitely with time since my dataset started so that my model can use it to differentiate the different time periods and adapt its prediction on recent observations --> The issue here is that most models are not able to extrapolate so I am not sure how it will behave on values of "time_since_start" that it never saw.

Have you ever encounter this usecase ? How did you deal with it ?

Answer 1

periods where it is high and periods where it is low

Some latent variable that slowly varies with time is affecting the base rate of your response variable. It would be best to directly observe that variable and add it to the dataset.

Second best approach would be to pick some appropriate window size (a kernel bandwidth), compute a moving average, and add that rate feature to the dataset where your model can observe it and learn the patterns. Also supply the rate feature at test + inference time, through imputation if necessary.

Answer 2

More data is not always better.
How do you know it is unsatisfactory to not consider older data? In your case it might actually improve the performance of your model. I would pefer one of the first two suggested methods, and see what their effect is on your performance indicator(s) compared to the time-naive baseline.

As a side note, do you have any clue what could cause this time-dependent shift in the outcome variable? The fact that it is not seasonal or explainable with your features probably means that it is due to some extraneous factor(s). Not having this (apparently important) data is a serious limitation to your model from a theoretical point of view.
After identifying this missing data, you might also think of a way of how you could acquire it.