I am experimenting with 3 years time series electrical demand data (kW) for a building and attempting to create regression supervised ML models from sci kit learn regressor algorithms but I have very poor performance (very high mean squared error). I have a GitHub Gist of the entire IPython notebook here.
There isn't a lot of wisdom here (and I don't have anyone to consult with) for what I am doing other than I know there is well developed analytic software (demand forecasting) that the power consulting industry uses and I am just attempting to recreate from scratch on own experimentation methods in Python.
The data that I am processing look like this below all recorded in 15 minute intervals.
0 2011-03-01 00:15:00 171.36
1 2011-03-01 00:30:00 181.44
2 2011-03-01 00:45:00 175.68
3 2011-03-01 01:00:00 180.00
The distribution of the kW data looks like this pic below which doesn't appear to have a bell shaped curve: (Could this be a poor performance reason?)
EDIT rolling average plot
Also in my experimentation I am adding in additional Python Pandas dataframes to represent the integer value of the time stamp 'day of the week', hour, minute, and month; where logically I am know electrical demand fluctuates greatly depending on these variables. These are some scatter plots below of the data compared to kW. (which maybe screwing everything up) For example the first scatter below is the hour of the day which is typical for buildings that the electrical demand increases during a typical work day. The outliers are most likely extreme weather conditions causing high demand where I do not have any weather data incorporated here...
In python if I do a
Ultimately I am hoping someone can give me some tips on why the model is horrible but maybe its just due to not enough data and/or strategy... Another person I have been questioning uses a clustering unsupervised learning approach but that doesn't make any sense to me...
Machine learning mastery also has a mini course and a large book I could purchase on time series forecasting. Is this more of a statistics approach? Does it require more 'normal' bell shaped distribution of the data?
Any tips to try or avenues to march down is greatly appreciated :)
EDIT GitHub gist was updated for a rolling average of the data as well as distribution column plot of kW data