I am fitting a binary classifier and I observe a temporal trend in the response variable, meaning that the actual percentage of positives fluctuates with time, I can see periods where it is high and periods where it is low. Note that most of my variables are not time related and that this fluctuation does not seem to be due to a change of mix in other variables. And this trends does not seem to be seasonal or cyclical either, so I don't want to use a Time Series model. I would like my model to know this so that its average prediction on new data points is more or less the average percentage of positives observed on recent points. I thought of doing this in several ways, but none of them are satisfactory:
- Train my model only on recent data (e.g. less than 6 month old) --> but then I am possibly missing a lot of information only contained in old data
- Assign a bigger importance to newer observations, by artificially including more of recent observations or by using the sample_weights setting of ML libraries (available on sklearn and XGboost) --> But it has basically the same disadvantage as the previous method. Old information is potentially lost
- Create a new variable "time_since_start" that grows indefinitely with time since my dataset started so that my model can use it to differentiate the different time periods and adapt its prediction on recent observations --> The issue here is that most models are not able to extrapolate so I am not sure how it will behave on values of "time_since_start" that it never saw.
Have you ever encounter this usecase ? How did you deal with it ?