Currently we have 10 thoundands log messages in the source codes which is maintained by different groups.

Some messages may be similar, but the sentence is not the same. My purpose is to create a centralized file which contains the unified message for reference.

For example:

printf("Failed to open the file %s", file)

printf("%s: failed to open file %s", TIMESTAMP, file)

printf("Failed to create the file %s")

I can extract all the message body which is in bold.They are not strict English sentence. but manually categorizing them takes too much time.

Is there a way to automatically categorizing the logs? I just want to reduce the manual work.

  • $\begingroup$ if you do not want any model, you can try with cosine similarity between two rows of data with some criteria. similar kind of data can be filtered $\endgroup$ – Akhilesh_IN Feb 26 '19 at 12:23

A simple idea, non-supervision: 1.Stemming, reduceing punctuations. Then you can use Bag of word with TF-IDF to represent the log sentences.

  1. Run K-means to cluster your sentences. If your sentences are belong to 5 type. You can set K=5.

Of course, you can use Regular Express to classify those sentences.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.