I have a large dataset containing execution logs from various tests conducted over several years. The logs can be noisy and often contain a plethora of messages detailing the ongoing activities during the test. Most messages are formatted as follows:
COMPONENT > [TIMESTAMP] [PHASE_OF_TEST] LOG_LEVEL_IF_ANY : LOG_MESSAGE
From these logs, I've managed to extract and aggregate metrics, obtaining the following information for each test run:
- Test name
- Test run date
- Number of OK, KO, NOT_EXECUTED test results
- Number of FATAL, ERROR, WARNING messages
- Duration of the test run
My goal is to develop a system that can automatically prioritize these tests based on metrics captured in the logs.
I aim to prioritize tests in such a way that those exhibiting unusual behavior or patterns (e.g., a sudden drop in OK results, high variance in execution time), or tests that haven't been run for some time, are flagged or categorized as more "relevant".
I'm undecided between using regression or classification methods. While regression could yield a "relevance" score to rank the tests, classification might offer a simpler, more interpretable model
The challenges and the advice I seek
I wonder whether a machine learning approach would be appropriate for this task. While I have been exposed to simple machine learning algorithms in the past, I find it challenging to apply in this situation due to the absence of a clear dependent variable indicating a test's "relevance". This makes it difficult to evaluate the performance of a predictive model.
I'm relatively new to statistics and machine learning and I seek guidance on what approach or techniques I should explore to construct such a system.
Any suggestions on how to deal with the absence of a "relevance" metric and which statistical methods or machine learning models would be most suited for this task, would be highly appreciated.