We are looking to predict counterfactual states from time-series data. In our problem we are looking to determine the energy savings from a grid-installed device that is varied on and off for many substations and feeders. Our methodology (open to critique) is to fit a model on hourly load for many substations and feeder's data of a years, then nudge the feature related to the device being on or off to get energy savings for a full timeperiod. So we
- train the model on all the existing data (kW ~ x1 + x2 + ...)
- make a prediction for the device being off at all timestamps (producing kW at every timestamp)
- make a prediction with the device on for all timestamps (producing kW at every timestamp)
- Subtract the "off" predictions with the "on" predictions.
While a lot of this is up in the air, my main question right now is related to lag terms and hyperparameter tuning. Typically if we are predicting the future we aim to have no information from future timesteps in the train data during hyperparamter tuning. We include lagged weather terms from the past. Some of us think that we would not be able to include lagged features in our training set unless the observations they refer to are actually in the training set, otherwise information is leaking. Is this concern sound?
Thoughts on how you would approach this and reasoning?
More info: we are using random forest and xgboost. Did this n the past with k-fold CV, but reassessing.