I'm sorry for asking a bit outlier topic question.

Currently, I'm trying to understand log parser processing and log analysis for anomaly detection in the ML/DL class. There are so many references for this (DeepLog, LogBERT, Loglizer, and so on..)

But one thing I cannot erase one thing in my head is " What is the actual purpose of real-time anomaly detection based on system log?"

According to my understanding, anomaly detection basically gives to support management ( anomaly avoiding or suggesting the anomaly reason and detailed info). and brief workflow is, the model trained the log which log sequence is normal behavior or not. and using that model we can judge whether real-time incoming logs will be an anomaly.

But basically, the system log is printed after something happens. It means real-time anomaly detection seems no need to avoid anomalies because there is no chance to do something in a time gap between the system log and the model doing some action. If this is correct, then we cannot avoid the error or warning and also we don't have enough time to solve issues I think.

As far as I know, Industry (IBM, other cloud companies) uses this kind of anomaly detection system in order to manage the system. But I'm wondering what is the exact role of real-time anomaly detection based on the system log.

  • 1
    $\begingroup$ It is quite useful to let the sysadmins know that their system is failing so they can decide what to do, e.g. restart a subsystem. Detecting anomalies in real-time seems quite useful to achieve that. $\endgroup$
    – noe
    Jun 20, 2023 at 8:43


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.