0
$\begingroup$

can you help me understand this better? I need to detect anomalies so I am trying to fit an lstm model using validation_data but the losses does not converge. Do they really need to converge? Does the validation data should resemble train or test data or inbetween? Also, which value should be lower, loss or val_loss ? Thankyou!

$\endgroup$

1 Answer 1

0
$\begingroup$

When validating machine learning models, you have to use a validation procedure that is consistent with your problem. For an anomaly detection use-case, it means correctly split your data, evaluate your model and with the right metrics.

Split of the data

You have to correctly choose the way you are splitting your data. By default, you have to define three different sets : training, validation and the test sets.

The train-validation-test split is the most appropriate if the observations are well independent and the notion of time is not important in your problem. It is the best one because the distribution of your training data should be similar to your validation and test datasets.

Exemple 1 : To detect anomalies in banking transactions, the observations are independent and time is not important. Train-validation-test split seems to be an appropriate choice.

Exemple 2 : To detect anomalous temperatures in time series, time is an important variable because it might be possible to learn these anomalous temperatures from future data, which would then introduce a look-forward bias. In that situation, refer to sklearn TimeSeriesSplit. If you have few observations, you can also take a look on cross-validation.

Because you are using LSTM models which are designed for time series modeling, I guess you might be in the second configuration.

Which loss to minimize ?

You have every time to minimize the validation loss function. The correct model selection would be such as :

  1. Select a set of models and features to optimize :
  2. For each model :
  • Train the model on the train set.
  • Evaluate your model on the validation set.
  1. Select your best model according to your validation set metrics.
  2. Evaluate it one and only one time on the test set.

As you want to minimize the loss on the validation set, you don't especially need to converge on the training set. For example, in an overfitting situation, you can obtain a very low loss on the training set but a very high loss on the validation set due to overfitting. Test set metrics is your true compass.

Which metrics to use ?

For an anomaly detection use-case, you have to carefully choose your metrics.

For most use-cases, the accuracy metrics will be very bad as the distribution of your labels is imbalanced and the positive labels (the anomalies) normally are more important than the non-anomaly class.

You have to select a metrics appropriate with respect to the previous reasons + the problem you are searching to solve.

$\endgroup$
4
  • $\begingroup$ Hi @Theudbald, thankyou for you answer! You are correct, I have time series temperature data, but I made up the validation and test data, with future time and different values so I don't need to split the initial data. I did follow the steps you said, I thought the mae loss should converge. I am using regression metrics, but I can't get lower than 0.2 mae loss and mape and cosine have weird values though. The models find some of the anomalies but not all using the automatic threshold. Do I need to update manually the threshold based on my observations? I though this should be automatic. $\endgroup$ Commented Jun 2, 2022 at 19:19
  • $\begingroup$ Hi @warriorforce, it may comes from your model. Have your tried simple models like linear regression or auto-regressive models ? In my experience, LSTM can be pretty difficult to setup for a first try. Here is an other post I have answered on this LSTM problem. For anomaly detection, you also have some very simple algorithms such as DBscan. datascience.stackexchange.com/questions/111454/… $\endgroup$
    – Theudbald
    Commented Jun 3, 2022 at 6:08
  • $\begingroup$ Hi @Theudbald, regarding the LSTM, I observed there was some different output when I increased the volume of data because it was very low. I think I can't get other results for that set of data anyhow. Thankyou very much for the suggestion with DBSCAN, it let me also to other models, it was helpful. $\endgroup$ Commented Jul 27, 2022 at 18:32
  • $\begingroup$ Glad it may had helped you. If you consider the answers useful, could you please upvote or approve the answer ? Thanks. $\endgroup$
    – Theudbald
    Commented Jul 30, 2022 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.