Putting together a Keras MLP to predict whether a value will exceed a static percent threshold in the next 15 minutes. The incoming data is a rolling percentage which moves smoothly for the most part because the data comes in every few milliseconds and is windowed over some fixed number of minutes. So, when the data is already above the threshold, it tends to stay there for a while.
When putting the data through the NN, it gets high accuracy, but this seems to be due to it correctly predicting that when it is currently over the threshold, it will also be over the threshold at some point (the next point) in the next x timesteps. The usefulness in the model would be if it could accurately predict before it crosses the threshold.
1) current point is over threshold - 1/0 2) current point is AM - 1/0 3) current day is weekday - 1/0 4) current percentage - 0.0-1.0 5-9) average of percentages in past 1/5/10/20/30 minute - 0.0-1.0
1 if a point is over threshold at any point in time after now and before now+15 minutes
Features 5-9 are intended to capture the inertia of the current percentage.
From feature importance, it looks like the current value is heavily used, followed by whether or not its over threshold, followed by the rolling means in order of time. I am currently changing the NN architecture and number of epochs in order to increase f1 score. Should I remove features 1 and 4, or rework the label in order to increase the accuracy of the predictive ability before its actually over the threshold?