I am the beginner of Machine learning. The process is:
I have different log files (System log, MSSQL Server log, Linux log, MySQL Log, FTP log, IIS log).If any input is given, I will find out which type of log using machine learning technique. Each log has a different format. Some logs don't have structure format(Linux, MySQL log, FTP log). In my analysis, these are all implemented using KNN algorithm(machine learning). But I don't know how to implement this? please give any suggestion about this.

The log type format:


Jan  5 08:39:01 iei-Virtual-Machine CRON[48622]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))

MySQL-error log:

2018-01-05 10:55:20 18856 [Warning] Unsafe statement is written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ between master and the slave. Statement: CALL SubmitGetChangeDetectionInfo(@_SubmitGetChangeDetectionInfo_0

Event log:

IE038,System log,Error,20/11/2017 12:47:51 PM,TerminalServices-Printers,1111,None,Driver HP Deskjet 3520 series required for printer IEC057(Mahendran) Printer is unknown. Contact the administrator to install the driver before you log in again.

FTP log:

2018-01-04 00:00:01 INFOEVOL\EC 63346 
DataChannelOpened - - 0 0 c1df3130-60e6-4678-9dcd-39177cc60d06 -IISAppslog:
2017-11-06 03:25:16 GET /IEIAppsLogin.aspx 
param=OwjgKJLT+ikfpFnxYbvZS/QWXTFP4GEXmT+qM7TeXTMqi5D7DKexzYjZc3aJNB0x 90 - Mozilla/5.0+(Windows+NT+6.3;+Win64;+x64)+AppleWebKit/537.36+
(KHTML,+like+Gecko)+Chrome/61.0.3163.100+Safari/537.36 302 0 0 234

MSSQL Server:

01/10/2018 07:07:07,Logon,Unknown,Login failed for user 'ms SQL'. Reason: Could not find a login matching the name provided. [CLIENT:]

2 Answers 2


You first need to preprocess your text to convert it into features that can be consumed by a machine learning algorithm. Then you can try this algorithm (a classifier, in this case) with the known examples you have. Then you can use that trained model to recognise logs it hasn't seen before.

If you don't have a preference of language you'd like to use, python with the scikit-learn library would be a good start.

For your problem, work through this tutorial which is very similar to what you're trying to achieve: http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html


As Elias Strehle mentioned in the comments, this does not sound like a problem for machine learning.

You will be able to solve this much faster by identifying a few simple patterns that indicate which type of log you are looking at.

I can already see that lines start with date string followed by a few tokens which look different in each log. You can define regex patterns that will only match a certain type of log, for example:

^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2} will match your Linux-syslog

^\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\s+\d+\s will match your MySQL-error log

^[A-Z]+\d+, will match your Event log

^\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\s+\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} will match your FTP log

And ^\d{2}/\d{2}/\d{4} will match your MSSQL Server log.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.