I was assigned this task to analyze the server logs of our application which contains exception logs, database logs event logs etc. I am new to machine learning, we use Spark with elastic search and Sparks MLlib(or PredictionIO).An example of the desired result would be to be able to predict based on the exception logs gathered to be able to predict which user is more likely to cause the next exception and at which feature(and bunch of other stuff to keep track and improve optimization of the application).
I have successfully been able to ingest data from ElasticSearch into spark and create DataFrames and map the data needed. What I would like to know is how do I approach the Machine Learning aspect of my implementation. I've been through articles and papers that talk about Data preprocessing, training the data models and creating labels and then generating predictions.
The questions I have are
How do I approach transforming the exiting log data into numerical vectors which can be used to datasets to be trained.
What algorithms do I use to train my dataset(with the limited knowledge i've gathered the past couple days, i was thinking bout implementing linear regression, please suggest which implementation would be best)
Just looking for suggestions on how to approach this problem.