In my graduation project, I use sensors to collect power usage data for home appliances with 5 minutes intervals, I want to create an ML model that takes in a variable number of values (len(dataset)) and predicts a variable number of readings in the future (Forecast_horizon), it could be 10 days readings to forecast the next 20 days, or 30 days readings to forecast 365 days and so on.
in other words, I want to create this API function
def forecast(data, forecast_horizon)
outputs future readings the same size as forecast horizon.
I went for LSTM, here's my approach:
the function get N data points, It creates lagged samples of (LAG) size then uses these lagged samples to predict one value in the future:
X:[1 2 3 4] -> Y:[]
then I train the LSTM on this dataset. when It reaches the end of the dataset, it appends the predicted value into a new lagged sample : [1 2 3 Xnew] -> [[Xnew+1]]
The function uses these newly created samples to "climb on its predictions" until it reaches the end of the forecast horizon. [1 2 3 Xnew] -> [[Xnew+1]]
[2 3 Xnew Xnew+1] -> [[Xnew+2]]
notice that I didn't use sequence to sequence output because to me the length of the output is unknown (Forecast_horizon) and it's going to be too long, 20*(60/5), and so on.
Now the problem with this approach is that in the first 10 or 20 new predictions, the model outputs somewhat decent values, however, after the 20th reading, it degrades into a straight line.
I tried to include hyperparameter tuning with Talos in my pipeline, and use the last 0.3 samples as a validation set, but the search space is just too big to try all combinations and I didn't get anything good doing this. also this model is going to be hosted in a serverless function where there aren't many resources, so extensive hyperparameters search isn't an option.
I noticed that playing with the LAG parameter, the number of LSTM cells, the depth of the network, using dropouts or not, and other hyperparameters. yields different results. sometimes decent ones.
but again doing this manually is just too hard.
It's guaranteed that most eclectic appliances consumption is stationary. there will be no trend (or else you need to fix your appliance), although there might be some seasonality in the yearly consumption of some appliances such as the air conditioner (works more in summer and less in winter) it's just a set of patterns that get repeated periodically.
so my question is, how can I achieve this task. a function that takes n number of data points and returns the best possible forecasted (Forecast_horizon) number of data points? is LSTM good? is there any other better model?
I also want to understand the relationship between all these different factors, the shape of the data, the number of cells, the depth of the network, the size of the Lagged X sample, and so on.
here's my code LSTM code
here's my dataset Dataset