2
$\begingroup$

Putting together a Keras MLP to predict whether a value will exceed a static percent threshold in the next 15 minutes. The incoming data is a rolling percentage which moves smoothly for the most part because the data comes in every few milliseconds and is windowed over some fixed number of minutes. So, when the data is already above the threshold, it tends to stay there for a while.

When putting the data through the NN, it gets high accuracy, but this seems to be due to it correctly predicting that when it is currently over the threshold, it will also be over the threshold at some point (the next point) in the next x timesteps. The usefulness in the model would be if it could accurately predict before it crosses the threshold.

Features:

1) current point is over threshold - 1/0
2) current point is AM - 1/0
3) current day is weekday - 1/0
4) current percentage - 0.0-1.0
5-9) average of percentages in past 1/5/10/20/30 minute - 0.0-1.0

Label:

1 if a point is over threshold at any point in time after now and before now+15 minutes

Features 5-9 are intended to capture the inertia of the current percentage.

From feature importance, it looks like the current value is heavily used, followed by whether or not its over threshold, followed by the rolling means in order of time. I am currently changing the NN architecture and number of epochs in order to increase f1 score. Should I remove features 1 and 4, or rework the label in order to increase the accuracy of the predictive ability before its actually over the threshold?

$\endgroup$
1
$\begingroup$

Consider your problem as a binary classification. We have two kinds of prediction: raise an alarm (state is going to change), do not raise an alarm (state is going to stay the same).

Actual / Predicted   Alarm goes off      Alarm did not go off
State changed        True positive       False positive
State stayed         False negative      True negative

Note the true negatives are useless - we don't care if the state stays in the same spot (below/above threshold). Therefore, the accuracy is NOT important for this system. Do not try to minimize the wrong metric. This extends to "do not use ROC/AUC" (in your problem). You can use a PR curve, but be careful with those (no interpolation, no AUC PR, these are wrong/useless). Basic metrics that you could use are F-score, precision and recall, or a re-weighting of those (if the false alarm is less important than a missed alarm, for example).

We also would like to deal with a huge "lag", trying to predict many time periods before (predicting just before the change is less useful). For this reason, I'd suggest investigating models other than MLP, notably those based on recurrent neural networks, such as LSTM. Also consider using time series prediction instead of classification - the literature on the subject is extensive and matches your problem really well.

$\endgroup$
  • $\begingroup$ State change! I can immediately see how that will improve the target, and then I will likely need to handle a class imbalance given with current labels its about 60/30 split over 5MM rows. My first approaches were time series regressions, but that was using the raw inputs and not this smoothed out rolling percentage, so I may return to see how that approach works as well. $\endgroup$ – user4446237 Oct 5 '17 at 18:06
  • $\begingroup$ Going to stick with this classification for now. The label change was simple (any point over threshold should be a 0), is 'binary_crossentropy' still a valid loss function? I am also keeping track of f1 score for CV. $\endgroup$ – user4446237 Oct 6 '17 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.