1
$\begingroup$

I have a dataset of logs with a specific structured format, and I'm looking for the best approach to detect anomalies within this data. I've already experimented with autoencoders and clustering techniques, but I'm curious if there are basic NLP methods or more advanced NLU techniques that could be more effective for this type of data.

Here's a sample of the log format:

MODULE DATE TIME_STAMP ACTION_TYPE META_INFO EXTRA_INFO
AppCreator 2019-06-15 2019-06-15 13:15:10 UIResponse {"segment": "App Branch", "tag": "Order", "activity": "MenuOpened", "version": "0.2", "view": "Primary Screen"} N/A
AppCreator 2019-06-15 2019-06-15 13:30:45 SysActivity {"context": "Layout", "tag": "Tab Shifted", "source": "", "element": "AppTabs", "activity": "Modify", "version": "0.2"} ["2"]

The DOMAIN_DATA field contains structured data in JSON format, and the CUSTOM_DATA field can either be a list or more complex structured data.

MODULE DATE TIME_STAMP ACTION_TYPE META_INFO EXTRA_INFO
AppCreator 2019-06-15 2019-06-15 13:15:10 UIResponse {"segment": "App Branch", "tag": "Order", "activity": "MenuOpened", "version": "0.2", "view": "Primary Screen"} N/A
AppCreator 2019-06-15 2019-06-15 13:30:45 SysActivity {"context": "Layout", "tag": "Tab Shifted", "source": "", "element": "AppTabs", "activity": "Modify", "version": "0.2"} ["2"]
AppCreator 2019-06-15 2019-06-15 13:45:30 UIAction {"segment": "App Design", "tag": "Submit", "activity": "ButtonPressed", "version": "0.3", "view": "Secondary Screen"} ["FormA", "ActionB", "RedirectC"]

I've also come across methods that involve log parsing using LLMs followed by anomaly detection, but I'm unsure if that's the right direction for this dataset.

Could anyone provide insights or suggestions on how to effectively leverage NLP/NLU for anomaly detection in this context? Any guidance or references?

Here's an extended table with 10 samples to help you get an understanding of what my input looks like for doing anomaly detection on the logs. This table showcases various interactions, system activities, and user actions within the application, providing a holistic view of the data:

To give readers a more generalistic understanding of the data, here's a sample of the log entries:

MODULE DATE TIME_STAMP ACTION_TYPE META_INFO EXTRA_INFO
AppCreator 2019-06-15 2019-06-15 13:15:10 UIResponse {"segment": "App Branch", "tag": "Order", "activity": "MenuOpened", "version": "0.2", "view": "Primary Screen"} N/A
AppCreator 2019-06-15 2019-06-15 13:30:45 SysActivity {"context": "Layout", "tag": "Tab Shifted", "source": "", "element": "AppTabs", "activity": "Modify", "version": "0.2"} ["2"]
AppCreator 2019-06-15 2019-06-15 13:45:30 UIAction {"segment": "App Design", "tag": "Submit", "activity": "ButtonPressed", "version": "0.3", "view": "Secondary Screen"} ["FormA", "ActionB", "RedirectC"]
AppCreator 2019-06-15 2019-06-15 14:00:00 SysActivity {"context": "Data", "tag": "DataSaved", "source": "FormB", "element": "Database", "activity": "Save", "version": "0.4"} ["DataX", "DataY"]
AppCreator 2019-06-15 2019-06-15 14:15:15 UIResponse {"segment": "App Settings", "tag": "Config", "activity": "MenuAccessed", "version": "0.2", "view": "Config Screen"} N/A
AppCreator 2019-06-15 2019-06-15 14:30:30 UIAction {"segment": "App Design", "tag": "Drag", "activity": "ElementDragged", "version": "0.3", "view": "Design Screen"} ["ElementZ", "PositionP"]
AppCreator 2019-06-15 2019-06-15 14:45:45 SysActivity {"context": "Layout", "tag": "LayoutChanged", "source": "User", "element": "LayoutGrid", "activity": "Update", "version": "0.5"} ["Layout1", "Layout2"]
AppCreator 2019-06-15 2019-06-15 15:00:00 UIResponse {"segment": "App Preview", "tag": "Preview", "activity": "PreviewOpened", "version": "0.2", "view": "Preview Screen"} N/A
AppCreator 2019-06-15 2019-06-15 15:15:15 UIAction {"segment": "App Feedback", "tag": "Feedback", "activity": "FeedbackGiven", "version": "0.4", "view": "Feedback Screen"} ["Positive", "FeatureSuggestion"]
AppCreator 2019-06-15 2019-06-15 15:30:30 SysActivity {"context": "Integration", "tag": "API", "source": "ExternalService", "element": "APIEndpoint", "activity": "Call", "version": "0.6"} ["API1", "Response200"]

What have I already tried?

I have already explored various unsupervised learning methods for anomaly detection on this dataset. Specifically, I've tried:

  • KMeans Clustering
  • Autoencoders
  • DBSCAN
  • Isolation Forests

In addition to these, I've also delved into deep learning methods, particularly a method called DeepLog. For those unfamiliar, DeepLog is a deep neural network model designed for anomaly detection in log files. I've read and implemented insights from the DeepLog paper (this is a survey paper the original paper of deeplog is here, wanted to try deep learning methods and thought of giving this a shot. However this still requires encoding the ACTION_TYPE and more importantly for the anomalies to be already labelled which isn't the case in my dataset.) and tried to adapt it to my dataset.

However, I'm still looking for more advanced or nuanced methods that might be better suited for the specific nature and format of my logs.


To continue or help give some direction into how I am thinking about solving this I thought it best to share my way of coming up with a solution -

Here's a step by step description of what I am doing in the implementation given below :

  1. Initialize a list called ents to store named entities.
  2. Initialize a list called pos_tags to store part-of-speech tags.
  3. Initialize a list called deps to store syntactic dependencies.
  4. Initialize a float variable called avg_sentiment to store the average sentiment score.
  5. Initialize a list called key_phrases to store key phrases.
  6. Split the input text into individual words using the word_tokenize() function.
  7. Remove stop words and punctuation from the tokenized text using lists comprehension.
  8. Lemmatize the remaining words using the lemmatizer.lemmatize() function.
  9. Perform named entity recognition on the lemmatized text using the nlp() function and store the results in ents.
  10. Perform part-of-speech tagging on the lemmatized text using the nlp() function and store the results in pos_tags.
  11. Perform syntactic dependency parsing on the lemmatized text using the nlp() function and store the results in deps.
  12. Calculate the average sentiment score of the text using the SentimentIntensityAnalyzer() class and store it in avg_sentiment.
  13. Extract key phrases from the text using the keywords() function and store them in key_phrases.
import json
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from spacy import displacy
from spacy.lang.en import English
from spacy.models import load_pretrained
from vader import SentimentIntensityAnalyzer
from gensim.summarization.keypoints import keywords

nlp = English()
stop_words = set(stopwords.words('english'))
lemmatizer = WordNetLemmatizer()

def process_text(text):
    data = json.loads(text)
    components = data["COMPONENT"]
    event_date = data["EVENT_DATE"]
    event_timestamp = data["EVENT_TIMESTAMP"]
    event_type = data["EVENT_TYPE"]
    domain_data = data["DOMAIN_DATA"]
    custom_data = data["CUSTOM_DATA"]

    tokens = word_tokenize(components)
    tokens = [t for t in tokens if t.isalpha() and t not in stop_words]
    tokens = [lemmatizer.lemmatize(t) for t in tokens]
    ents = nlp(tokens)
    entities = [entity.text for entity in ents if entity.label_ == 'PEOPLE']
    pos_tags = nlp(tokens, disable=['ner'])
    pos_tags = [t.pos_ for t in pos_tags]
    parse_tree = nlp(tokens, disable=['ner', 'pos'])
    dependencies = [t.dep_ for t in parse_tree]
    sentiments = []
    for sentence in tokens:
        sentiments.append(SentimentIntensityAnalyzer().polarity_scores(sentence))
    avg_sentiment = np.mean([s[0] for s in sentiments])
    doc_frequency = keywords(text, density=0.5)
    key_phrases = [t for t, freq in doc_frequency.items() if freq > 0.5]

return {
    'entities': entities,
    'pos_tags': pos_tags,
    'dependencies': dependencies,
    'avg_sentiment': avg_sentiment,
    'key_phrases': key_phrases
}

text = """COMPONENT = ServiceStudio,    EVENT_DATE = 2019-07-02,    EVENT_TIMESTAMP = 2019-07-02 14:20:11,  EVENT_TYPE = UIEvent, DOMAIN_DATA = "{
  ""area"": ""ESpace Tree"",
  ""label"": ""Product"",
  ""type"": ""ContextMenuOpened"",
  ""ver"": ""0.1"",
  ""window"": ""Main Window""
}"                                                              
, CUSTOM_DATA    =           "[
  {
    ""AppName"": ""teste"",
    ""AppType"": 1,
    ""ModuleType"": 1,
    ""Name"": ""teste""
  }
]"
"""
result = process_text(text)
print(result)

Input: I want to use the above nlp methods to understand the logs written in this format (extracted from a table with a similar format as the one I already have shared above) -

MODULE = AppCreator, EVENT_DATE = 2019-07-02, EVENT_TIMESTAMP = 2019-07-02 14:20:11, EVENT_TYPE = UIEvent, DOMAIN_DATA = "{
""area"": ""ESpace Tree"", ""label"": ""Product"", ""type"": ""ContextMenuOpened"", ""ver"": ""0.1"", ""window"": ""Main Window"" }" , CUSTOM_DATA = "[ { ""AppName"": ""teste"", ""AppType"": 1, ""ModuleType"": 1, ""Name"": ""teste"" } ]"

Output:

{ 'entities': ['AppCreator'], 'pos_tags': ['NNP'],
'dependencies': [ {'rel': 'det', 'head': 'UIEvent', 'deps': ['Component']}, {'rel': 'nsubjpass', 'head': 'UIEvent', 'deps': ['Area']}, {'rel': 'obj', 'head': 'ContextMenuOpened', 'deps': ['Label']} ], 'avg_sentiment': 0.0, 'key_phrases': ['ServiceStudio', 'UIEvent', 'Espace Tree', 'Product', 'ContextMenuOpened', 'Main Window'] }

Now with this information, I am looking to figure out possible ways in which an anomaly can be detected in the log.

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.