2
$\begingroup$

I am working on the time series classification task that focuses on predicting a fault. I framed the problem as a multi-step forecasting problem, where my goal is to predict to the class at y(t+1 hour) = f(X_lagged). My target y is either 1, or 0, and the problem is essentially a binary classification problem. In the papers I have read, recall/precision/f1 metrics are used as a model evaluation metric. However, if I want to state the problem as "What is the probability that the fault will happen in 1 hour?", what metric do I need to use?

$\endgroup$
0
$\begingroup$

[edited]

The Area Under the Curve (AUC) is a good fit for this kind of problem, but it's not exactly a standard case. The ROC curve is built by ranking your instances by their predicted probability of fault, for example:

p(faulty)   gold
0.03        0
0.14        1
0.19        0
0.23        0
0.31        1
0.32        0
0.65        1
0.78        1
0.83        0
0.90        1

Each level of probability corresponds to a possible threshold for the binary classification problem. The number of True/False Positive/Negative can be calculated for every possible level of threshold:

p(faulty)   gold TP TN FP FN
                 5  0  5  0
0.03        0    5  1  4  0
0.14        1    4  1  4  1   
0.19        0    4  2  3  1 
0.23        0    4  3  2  1
0.31        1    3  3  2  2
0.32        0    3  4  1  2
0.65        1    2  4  1  3
0.78        1    1  4  1  4
0.83        0    1  5  0  4
0.90        1    0  5  0  5

At this stage one obtains exactly the data required for a standard ROC curve: from these values precision and recall scores can be computed for every level and plotted on the ROC curve. The AUC is used in the usual way.

This method takes into account the continuity of the probabilities while matching predictions against binary classes. An example of problem where this evaluation method is used: https://pan.webis.de/clef15/pan15-web/author-identification.html

$\endgroup$
  • $\begingroup$ It won't work because the dataset is imbalanced. $\endgroup$ – eemamedo Aug 1 at 13:14
  • $\begingroup$ I don't see why it wouldn't work: typically in an imbalanced binary problem one would use precision/recall/f-score based on the minority class. it seems to me that you could do the same with AUC here... but maybe I'm missing something? $\endgroup$ – Erwan Aug 1 at 13:23
  • $\begingroup$ I just don't see how it ROC-AUC can be used to answer the question that the fault will happen in 1 hour. The value I got is the same as F1-macro average between two classes and equal to 0.86, where the value for normal operations is 0.99 and for faulty operation is 0.74 (both for recall). $\endgroup$ – eemamedo Aug 1 at 13:33
  • $\begingroup$ Sorry I realized that I should have given more detail in my answer, see my edits. $\endgroup$ – Erwan Aug 1 at 14:19
0
$\begingroup$

I had same cases in my previous projects. I prefer custom solution for this:

First: You have imbalanced class dataset, try to use SMOTE or other up/down sampling techniques. After that (or without doing that) you should focus on your cost of prediction.

For example if you have an alarm system for a factory and you create an alarm which will stop the production and it costs 1M dollars, and do not create signal and its FN and it costs 100M dollars, then you should integrate your false prediction and true prediction costs to your model evaluation.

You should write your custom grid search with costs and tune your parameters with different thresholds (based on predict_proba).

If you had a balanced class dataset I would advice auc-roc and then calculate costs but in your case you have to prioritize costs.

I hope its clear.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.