1
$\begingroup$

The data

I have a large dataset containing execution logs from various tests conducted over several years. The logs can be noisy and often contain a plethora of messages detailing the ongoing activities during the test. Most messages are formatted as follows:

COMPONENT > [TIMESTAMP] [PHASE_OF_TEST] LOG_LEVEL_IF_ANY : LOG_MESSAGE

From these logs, I've managed to extract and aggregate metrics, obtaining the following information for each test run:

  • Test name
  • Test run date
  • Number of OK, KO, NOT_EXECUTED test results
  • Number of FATAL, ERROR, WARNING messages
  • Duration of the test run

The goal

My goal is to develop a system that can automatically prioritize these tests based on metrics captured in the logs.

I aim to prioritize tests in such a way that those exhibiting unusual behavior or patterns (e.g., a sudden drop in OK results, high variance in execution time), or tests that haven't been run for some time, are flagged or categorized as more "relevant".

I'm undecided between using regression or classification methods. While regression could yield a "relevance" score to rank the tests, classification might offer a simpler, more interpretable model

The challenges and the advice I seek

I wonder whether a machine learning approach would be appropriate for this task. While I have been exposed to simple machine learning algorithms in the past, I find it challenging to apply in this situation due to the absence of a clear dependent variable indicating a test's "relevance". This makes it difficult to evaluate the performance of a predictive model.

I'm relatively new to statistics and machine learning and I seek guidance on what approach or techniques I should explore to construct such a system.

Any suggestions on how to deal with the absence of a "relevance" metric and which statistical methods or machine learning models would be most suited for this task, would be highly appreciated.

$\endgroup$

1 Answer 1

1
$\begingroup$

Welcome to Data Science!

The first step is to make clear for yourself and future models the output you are looking for. It appears it's clear in your mind which tests to priorities but it's not in the data as a column.

You rightly point out that you could generate a rank or some priority metric. Once you have that, then you can consider using an ML model. So step 1 is generate your desired output that is meaningful for the task.

As a first step, I would consider creating a priority column for each test with three categories representing High, Medium and Low. You have to then decide how to generate these categories from your test data. An algorithm would be best but in that case you wouldn't need a model. Alternatively, use your judgement or realistically a combination of both.

Once you have your input data (features) and your assigned output (in this case 3 classes) you can follow the usual ML approach.

You can then extend this with additional discrete labels or alternatively switch to a numeric score and use regression type models if you need additional fidelity.

$\endgroup$
3
  • $\begingroup$ Thank you for your insightful and easily understandable response, which I appreciate as I am not yet an expert in the domain. The idea of creating a priority column is appealing, but the challenge lies in the sheer volume of tests that would need manual labeling. Could you suggest a way forward? Should I attempt to develop a heuristic formula with selected coefficients for each input variable to produce a preliminary priority score? If so, would machine learning still add value? Or is labeling just a subset of the tests a viable alternative? Or is there some unsupervised approach to labeling? $\endgroup$ Oct 27, 2023 at 7:32
  • $\begingroup$ Labelling is the most tedious but sadly the most valuable exercise in ML. There isn't really a way round it but you do need an output. $\endgroup$
    – fswings
    Oct 27, 2023 at 10:36
  • $\begingroup$ Using an unsupervised approach (the most common is clustering) is about finding patterns in the data. It can't guess what's important to you. $\endgroup$
    – fswings
    Oct 27, 2023 at 10:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.